Skip to main content

README

· One min read
caution

This project page is still under migration. Not all blogs are here.

Please go to https://huakunshen.super.site/ to see a full list of my blogs.

iife

· 3 min read

title: IIFE authors: [huakun] tags: [Web]

https://developer.mozilla.org/en-US/docs/Glossary/IIFE

An IIFE (Immediately Invoked Function Expression) is a JavaScript function that runs as soon as it is defined.

Immediately Invoked Function Expression (IIFE) 是 JavaScript 中的一种常见的设计模式，它是一个立即执行的匿名函数表达式。IIFE 通常用于创建一个独立的作用域，避免变量污染全局作用域。

(function () {
  // …
})();

(() => {
  // …
})();

(async () => {
  // …
})();

IIFE is usually used to create a separate scope to avoid polluting the global scope. In this blog I will instead talk about the use of IIFE in hosting web pages.

Traditionally, static websites are compiled into an index.html file as an entrypoint. css and js files are linked in the head and body tags.

Here is an example of a simple index.html file:

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <link rel="icon" type="image/svg+xml" href="/vite.svg" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>Vite + Vue + TS</title>
    <script type="module" crossorigin src="/assets/index-D7F47PqG.js"></script>
    <link rel="stylesheet" crossorigin href="/assets/index-DRBiz0Jz.css" />
  </head>
  <body>
    <div id="app"></div>
  </body>
</html>

You can see that in such a client-side rendered web page, the HTML file doesn't contain any content. The content is rendered by the JavaScript file linked in the head tag.

The index.html is only used as a container for the JavaScript file. The JavaScript file is responsible for rendering the content of the web page.

Theoretically, we can ship a single JavaScript file contains all the logic and content of the web page (including styles and images (png doesn't work, but svg does)).;

Vite Config​

For example, this is the original vite.config.ts file for a vue project:

import { defineConfig } from "vite";
import vue from "@vitejs/plugin-vue";

// https://vitejs.dev/config/
export default defineConfig({
  plugins: [vue()],
});

Output dist folder:

dist
├── assets
│   ├── Vue.js_Logo_2.svg-BtIZHRhy.png
│   ├── index-CjgLCVzZ.css
│   └── index-CwnRthTM.js
├── index.html
└── vite.svg

Next, set output format to iife

import { defineConfig } from "vite";
import vue from "@vitejs/plugin-vue";
import path from "path";

// https://vitejs.dev/config/
export default defineConfig({
  plugins: [vue()],
  build: {
    emptyOutDir: false,
    rollupOptions: {
      input: path.resolve(__dirname, "./src/main.ts"),
      output: {
        format: "iife",
        dir: path.resolve(__dirname, "./dist"),
        entryFileNames: "web.js",
      },
    },
  },
});
dist
├── assets
│   └── Vue.js_Logo_2.svg-BtIZHRhy.png
├── vite.svg
└── web.js

Now how to use the web.js without an index.html file?

Let's serve the dist folder with a simple http server:

serve dist --cors

The web.js is at http://localhost:3000/web.js

In an HTML file,

<!DOCTYPE html>
<html lang="en">
  <body>
    <iframe></iframe>
    <script>
      fetch("http://localhost:3000/web.js", {
        method: "GET",
      })
        .then((res) => res.text())
        .then((data) => {
          document
            .querySelector("iframe")
            .contentDocument.write(
              "<div id='app'/><script>".concat(data, "<\/script>")
            );
        });
    </script>
  </body>
</html>

Here we are rendering the content of web.js in an iframe. It can also render the page directly as long as there is a <div id="app" />.

<!DOCTYPE html>
<html lang="en">
  <body>
    <script>
      fetch("http://localhost:3000/web.js", {
        method: "GET",
      })
        .then((res) => res.text())
        .then((data) => {
          document.write(
            "<div id='app'></div><script>".concat(data, "<\/script>")
          );
        });
    </script>
  </body>
</html>

This is because the main.ts is like this

import { createApp } from "vue";
import "./style.css";
import App from "./App.vue";

createApp(App).mount("#app");

So what can be the use case of this?

  • The key benefit is that the website content doesn't have be to hosted on a server. It can be saved in a database and fetched by the client like a function.

Tauri Plugin System Design

· 8 min read
Huakun Shen
Huakun Shen
Website Owner

In Raycast Analysis and uTools Analysis I discussed the two successful app launchers and their plugin system designs. But both of them have big limitations. Raycast is mac-only. uTools is cross-platform (almost perfect), but it is built with Electron, thus large bundle size and memory consumption.

Tauri is a new framework that can build cross-platform desktop apps with Rust and Web. With much smaller bundle size and memory consumption. It’s a good choice for building a cross-platform app launcher.

Requirements​

  • Plugins can be built with JS frontend framework, so it’s easier for develop to build
  • UI can be controlled by plugin
  • Sandbox preferred, never trust plugins not developed by official team, community plugin could be malicious. Neither of Raycast, Alfred, uTools used sandbox. So we can discuss this as well.

Solution​

Plugins will be developed as regular single page application. They will be saved in a directory like the following.

plugins/
├── plugin-a/
│   └── dist/
│       ├── index.html
│       └── ...
└── plugin-b/
    └── dist/
        ├── index.html
        └── ...

Optionally use symbolic link to build the following structure (link dist of each plugin to the plugin name. You will see why this could be helpful later.

plugins-link/
├── plugin-a/
│   ├── index.html
│   └── ...
└── plugin-b/
    ├── index.html
    └── ...

When a plugin is triggered, the main Tauri core process will start a new process running a http server serving the entire plugins or plugins-link folder as static asset. The http server can be actix-web.

Then open a new WebView process

const w = new WebviewWindow('plugin-a', {
	url: 'http://localhost:8000/plugin-a'
});

If we didn’t do the dist folder symlink step, the url would be http://localhost:8000/plugin-a/dist

Do the linking could avoid some problem.

One problem is base url. A single page application like react and vue does routing with url, but the base url is / by default. i.e. index page is loaded on http://localhost:8000. If the plugin redirects to /login, it should redirect to http://localhost:8000/login instead of http://localhost:8000/plugin-a/login

In this case, https://vite-plugin-ssr.com/base-url, https://vitejs.dev/guide/build#public-base-path can be configured in vite config.

Another solution is to use proxy in the http server. Like proxy_pass in nginx config.

API​

Now the plugin’s page can be loaded in a WebView window.

However, a plugin not only needs to display a UI, but also need to interact with system API to implement more features, such as interacting with file system. Now IPC is involved.

Tauri by default won’t allow WebView loaded from other sources to run commands or call Tauri APIs.

See this security config dangerousRemoteDomainIpcAccess

https://tauri.app/v1/api/config/#securityconfig.dangerousremotedomainipcaccess

"security": {
	"csp": null,
	"dangerousRemoteDomainIpcAccess": [
		{
			"domain": "localhost:8000",
			"enableTauriAPI": true,
			"windows": ["plugin-a"],
			"plugins": []
		}
	]
},

enableTauriAPI determines whether the plugin will have access to the Tauri APIs. If you don’t want the plugin to have the same level of permission as the main app, then set it to false.

This not only work with localhost hosted plugins. The plugin can also be hosted on public web (but you won’t be able to access it if there is no internet). This will be very dangerous, as compromised plugin on public web will affect all users. In addition, it’s unstable. Local plugin is always safer.

There is another plugins attribute used to control which tauri plugin’s (The plugin here means plugin in rust for Tauri framework, not our plugin) command the plugin can call.

https://tauri.app/v1/api/config/#remotedomainaccessscope.plugins

plugins is The list of plugins that are allowed in this scope. The names should be without the tauri-plugin- prefix, for example "store" for tauri-plugin-store.

For example, Raycast has a list of APIs exposed to extensions (https://developers.raycast.com/api-reference/clipboard)

Raycast uses NodeJS runtime to run plugins, so plugins can access file system and more. This is dangerous. From their blog https://www.raycast.com/blog/how-raycast-api-extensions-work, their solution is to open source all plugins and let the community verify the plugins.

This gives plugins more freedom and introduces more risks. In our approach with Tauri, we can provide a Tauri plugin for app plugins with all APIs to expose to the extensions. For example, get list of all applications, access storage, clipboard, shell, and more. File system access can also be checked and limited to some folders (could be set by users with a whitelist/blacklist). Just don’t give plugin access to Tauri’s FS API, but our provided, limited, and censored API plugin.

How to give plugin full access to OS and FS?​

Unlike Raycast where the plugin is run directly with NodeJS, and render the UI by turning React into Swift AppKit native components. The Tauri approach has its UI part in browser. There is no way to let the UI plugin access OS API (like FS) directly. The advantage of this approach is that the UI can be any shape, while Raycast’s UI is limited by its pre-defined UI components.

If a plugin needs to run some binary like ffmpeg to convert/compress files, the previous sandbox API method with a custom Tauri plugin won’t work. In this scenario, this will be more complicated. Here are some immature thoughts

  • The non-UI part of the plugin will need a JS runtime if written in JS, like NodeJS or bun.js
  • Include plugin script written in python, Lua, JS… and UI plugin runs them using a shell API command (like calling a CLI command)
  • If the plugin need a long running backend, a process must be run separately, but how can the UI plugin communicate with the backend plugin? The backend plugin will probably need to be an http server or TCP server.
    • And how to stop this long running process?

Implementation Design​

User Interface​

Raycast supports multiple user interfaces, such as list, detail, form.

To implement this in Jarvis, there are 2 options.

  1. The extension returns a json list, and Jarvis render it as a list view.
  2. Let the extension handles everything, including list rendering.

Option 1​

Could be difficult in our case, as we need to call a JS function to get data, this requires importing JS from Tauri WebView or run the JS script with a JS runtime and get a json response.

To do this, we need a common API contract in JSON format on how to render the response.

  1. Write a command script cmd1.js

  2. Jarvis will call bun cmd1.js with argv, get response

    Example of a list view

    {
        "view": "list",
        "data": [
            {
                "title": "Title 1",
                "description": "Description 1"
            },
            {
                "title": "Title 2",
                "description": "Description 2"
            },
            {
                "title": "Title 3",
                "description": "Description 3"
            }
        ]
    }

This method requires shipping the app with a bun runtime (or download the runtime when app is first launched).

After some thinking, I believe this is similar to script command. Any other language can support this. One difference is, “script command” relies on users’ local dependency, custom libraries must be installed for special tasks, e.g. pandas in python. It’s fine for script command because users are coders who know what they are doing. In a plugin, we don’t expect users to know programming, and install libraries. So shipping a built JS dist with all dependencies is a better idea. e.g. bun build index.ts --target=node > index.js, then bun index.js to run it without installing node_modules.

https://bun.sh/docs/bundler

In the plugin’s package.json, list all commands available and their entrypoints (e.g. dist/cmd1.js, dist/cmd2.js).

{
    "commands": [
        {
            "name": "list-translators",
            "title": "List all translators",
            "description": "List all available translators",
            "mode": "cmd"
        }
    ]
}

Option 2​

If we let the extension handle everything, it’s more difficult to develop, but less UI to worry about.

e.g. translate input1 , press enter, open extension window, and pass the input1 to the WebView.

By default, load dist/index.html as the plugin’s UI. There is only one entrypoint to the plugin UI, but a single plugin can have multiple sub-commands with url path. e.g. http://localhost:8080/plugin-a/command1

i.e. Routes in single page app

All available sub-commands can be specified in package.json

{
    "commands": [
        {
            "name": "list-translators",
            "title": "List all translators",
            "description": "List all available translators",
            "mode": "cmd"
        },
        {
            "name": "google-translate",
            "title": "Google Translate",
            "description": "Translate a text to another language with Google",
            "mode": "view"
        },
        {
            "name": "bing-translate",
            "title": "Bing Translate",
            "description": "Translate a text to another language with Bing",
            "mode": "view"
        }
    ]
}

If mode is view, render it. For example, bing-translate will try to load http://localhost:8080/translate-plugin/bing-translate

If mode is cmd, it will try to run bun dist/list-translators and render the response.

mode cmd can have an optional language field, to allow using Python or other languages.

Script Command​

Script Command from Raycast is a simple way to implement a plugin. A script file is created, and can be run when triggered. The stdout is sent back to the main app process.

Supported languages by Raycast script command are

  • Bash
  • Apple Script
  • Swift
  • Python
  • Ruby
  • Node.js

In fact, let users specify an interpreter, any script can be run, even executable binaries.

Alfred has a similar feature in workflow. The difference is, Raycast saves the code in a separate file, and Alfred saves the code within the workflow/plugin (in fact also in a file in some hidden folder).

Reference​

Tauri Universal Build for Mac (Solve SSL Problem)

· 3 min read
Huakun Shen
Huakun Shen
Website Owner

I had problem building a universal Tauri app for Mac (M1 pro).

rustup target add x86_64-apple-darwin
rustup target add aarch64-apple-darwin

npm run tauri build -- --target universal-apple-darwin

The problem was with OpenSSL. The error message was:

    Finished `release` profile [optimized] target(s) in 4.50s
   Compiling openssl-sys v0.9.102
   Compiling cssparser v0.27.2
   Compiling walkdir v2.5.0
   Compiling alloc-stdlib v0.2.2
   Compiling markup5ever v0.11.0
   Compiling uuid v1.8.0
   Compiling fxhash v0.2.1
   Compiling crossbeam-epoch v0.9.18
   Compiling selectors v0.22.0
   Compiling html5ever v0.26.0
   Compiling indexmap v1.9.3
   Compiling tracing-core v0.1.32
error: failed to run custom build command for `openssl-sys v0.9.102`

Caused by:
  process didn't exit successfully: `/Users/hacker/Dev/projects/devclean/devclean-ui/src-tauri/target/release/build/openssl-sys-2efafcc1e9e30675/build-script-main` (exit status: 101)
  --- stdout
  cargo:rerun-if-env-changed=X86_64_APPLE_DARWIN_OPENSSL_LIB_DIR
  X86_64_APPLE_DARWIN_OPENSSL_LIB_DIR unset
  cargo:rerun-if-env-changed=OPENSSL_LIB_DIR
  OPENSSL_LIB_DIR unset
  cargo:rerun-if-env-changed=X86_64_APPLE_DARWIN_OPENSSL_INCLUDE_DIR
  X86_64_APPLE_DARWIN_OPENSSL_INCLUDE_DIR unset
  cargo:rerun-if-env-changed=OPENSSL_INCLUDE_DIR
  OPENSSL_INCLUDE_DIR unset
  cargo:rerun-if-env-changed=X86_64_APPLE_DARWIN_OPENSSL_DIR
  X86_64_APPLE_DARWIN_OPENSSL_DIR unset
  cargo:rerun-if-env-changed=OPENSSL_DIR
  OPENSSL_DIR unset
  cargo:rerun-if-env-changed=OPENSSL_NO_PKG_CONFIG
  cargo:rerun-if-env-changed=PKG_CONFIG_ALLOW_CROSS_x86_64-apple-darwin
  cargo:rerun-if-env-changed=PKG_CONFIG_ALLOW_CROSS_x86_64_apple_darwin
  cargo:rerun-if-env-changed=TARGET_PKG_CONFIG_ALLOW_CROSS
  cargo:rerun-if-env-changed=PKG_CONFIG_ALLOW_CROSS
  cargo:rerun-if-env-changed=PKG_CONFIG_x86_64-apple-darwin
  cargo:rerun-if-env-changed=PKG_CONFIG_x86_64_apple_darwin
  cargo:rerun-if-env-changed=TARGET_PKG_CONFIG
  cargo:rerun-if-env-changed=PKG_CONFIG
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_x86_64-apple-darwin
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_x86_64_apple_darwin
  cargo:rerun-if-env-changed=TARGET_PKG_CONFIG_SYSROOT_DIR
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR
  run pkg_config fail: pkg-config has not been configured to support cross-compilation.

  Install a sysroot for the target platform and configure it via
  PKG_CONFIG_SYSROOT_DIR and PKG_CONFIG_PATH, or install a
  cross-compiling wrapper for pkg-config and set it via
  PKG_CONFIG environment variable.

  --- stderr
  thread 'main' panicked at /Users/hacker/.cargo/registry/src/index.crates.io-6f17d22bba15001f/openssl-sys-0.9.102/build/find_normal.rs:190:5:


  Could not find directory of OpenSSL installation, and this `-sys` crate cannot
  proceed without this knowledge. If OpenSSL is installed and this crate had
  trouble finding it,  you can set the `OPENSSL_DIR` environment variable for the
  compilation process.

  Make sure you also have the development packages of openssl installed.
  For example, `libssl-dev` on Ubuntu or `openssl-devel` on Fedora.

  If you're in a situation where you think the directory *should* be found
  automatically, please open a bug at https://github.com/sfackler/rust-openssl
  and include information about your system as well as this message.

  $HOST = aarch64-apple-darwin
  $TARGET = x86_64-apple-darwin
  openssl-sys = 0.9.102


  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...
       Error failed to build x86_64-apple-darwin binary: failed to build app

Solution​

Install OpenSSL with brew

brew install openssl
export export OPENSSL_DIR=$(brew --prefix openssl)

This problem was not fully solved. I had to run GitHub Action on macos-13 runners (intel CPU). The build passes but the resulting app won't run on x86_64 Macs. Keeps saying the openssl lib cannot be loaded. I will update this post when I find a solution.

Read more here https://github.com/tauri-apps/tauri/issues/9684#event-12728702751

The real source of problem was actually git2's dependency (https://crates.io/crates/git2/0.18.3/dependencies) openssl-sys. Removing git2 from my app fixed all problems. Running on macos-14 runner (M1 pro) worked fine.

openssl-sys is a OpenSSL bindings for rust. So it doesn't include the actual OpenSSL library. You need to install OpenSSL on your system.

My guess is, during the build process on GitHub Action, the openssl library location is different from the one on my local machine, and the path is burned into the binary. So the binary won't run on other machines. This is just a guess. There must be some solution. I will update this post when I find it.

NestJS + Neo4j + GraphQL Setup

· 5 min read
Huakun Shen
Huakun Shen
Website Owner

GitHub Repo: https://github.com/HuakunShen/nestjs-neo4j-graphql-demo

I haven't found a good update-to-date example of using Neo4j with NestJS and GraphQL. So I decided to write one myself.

Neo4j's graphql library has updated its API, some examples I found online were outdated (https://neo4j.com/developer-blog/creating-api-in-nestjs-with-graphql-neo4j-and-aws-cognito/). This demo uses v5.x.x.

GraphQL Schema​

type Mutation {
  signUp(username: String!, password: String!): String
  signIn(username: String!, password: String!): String
}

# Only authenticated users can access this type
type Movie @authentication {
  title: String
  actors: [Actor!]! @relationship(type: "ACTED_IN", direction: IN)
}

# Anyone can access this type
type Actor {
  name: String
  movies: [Movie!]! @relationship(type: "ACTED_IN", direction: OUT)
}

# Only authenticated users can access this type
type User @authentication {
  id: ID! @id
  username: String!
  # this is just an example of how to use @authorization to restrict access to a field
  # If you list all users without the plaintextPassword field, you will see all users
  # If you list all users with the plaintextPassword field, you will only see the user whose id matches the jwt.sub (which is the id of the authenticated user)
  # in reality, never store plaintext passwords in the database
  plaintextPassword: String!
    @authorization(filter: [{ where: { node: { id: "$jwt.sub" } } }])
  password: String! @private
}

NestJS Server Configuration​

GraphQL Module​

A GraphQL module can be generated with bunx nest g mo graphql.

Here is the configuration. In new Neo4jGraphQL(), authorization key is provided for JWT auth. Queries can be restricted by adding @authentication or @authorization to the type.

One important thing to note is the custom auth resolvers. Neo4jGraphQL auto-generate types, queries, mutations implementations for the types in the schema to provide basic CRUD operations, but custom functions like sign in and sign up must be implemented separately. Either as regular REST endpoints in other modules or provide a custom resolver to the Neo4jGraphQL instance.

Usually in NestJS, you would add resolvers to the providers list of the module, but in this case, the resolvers must be added to the Neo4jGraphQL instance. Otherwise you will see the custom queries defined in schema in the playground, but they always return null.

@Module({
  imports: [
    GraphQLModule.forRootAsync<ApolloDriverConfig>({
      driver: ApolloDriver,
      useFactory: async () => {
        export const { NEO4J_URI, NEO4J_USERNAME, NEO4J_PASSWORD } =
          envSchema.parse(process.env);
        export const neo4jDriver = neo4j.driver(
          NEO4J_URI,
          neo4j.auth.basic(NEO4J_USERNAME, NEO4J_PASSWORD)
        );

        const typedefPath = path.join(RootDir, "src/graphql/schema.gql");
        export const typeDefs = fs.readFileSync(typedefPath).toString();

        const neoSchema = new Neo4jGraphQL({
          typeDefs: typeDefs,
          driver: neo4jDriver,
          resolvers: authResolvers, // custom resolvers must be added to Neo4jGraphQL instead of providers list of NestJS module
          features: {
            authorization: {
              key: "huakun",
            },
          },
        });

        const schema = await neoSchema.getSchema();
        return {
          schema,
          plugins: [ApolloServerPluginLandingPageLocalDefault()],
          playground: false,
          context: ({ req }) => ({
            token: req.headers.authorization,
          }),
        };
      },
    }),
  ],
  providers: [],
})
export class GraphqlModule {}

The resolver must be provided to Neo4jGraphQL constructor. It must be an object, so NestJS's class-based resolver won't work.

You must provide regular apollo stype resolvers. See https://neo4j.com/docs/graphql/current/ogm/installation/ for similar example.

export const authResolvers = {
  Mutation: {
    signUp: async (_source, { username, password }) => {
      ...
      return createJWT({ sub: users[0].id });
    },
    signIn: async (_source, { username, password }) => {
      ...
      return createJWT({ sub: user.id });
    },
  },
};

Read the README.md of this repo for more details. Run the code and read the source code to understand how it works. It's a minimal example.

Codegen​

https://the-guild.dev/graphql/codegen is used to generate TypeScript types and more from the GraphQL schema.

Usually you provide the graphql schema file, but in this demo, the schema is designed for neo4j and not recognized by the codegen tool.

You need to let Neo4jGraphQL generate the schema and deploy it to a server first, then provide the server's endpoint to the codegen tool. Then the codegen tool will introspect the schema from the server and generate the types.

Make sure the server is running before running codegen

cd packages/codegen
pnpm codegen

The generated files are in the packages/codegen/src/gql folder.

Sample operations can be added to packages/codegen/operations. Types and caller for operations will also be generated.

Read the documentation of codegen for more details.

Examples is always provided in the packages/codegen folder.

This is roughly how the generated code it works:

You get full type safety when calling the operations. The operations documents are predefined in a central place rather than in the code. This is useful when you have a large project with many operations. Modifying one operation will update all the callers. And if the type is no longer valid, the compiler will tell you.

The input variables are also protected by TypeScript. You won't need to guess what the input variables are. The compiler will tell you.

import { CreateMoviesDocument } from "./src/gql/graphql";

async function main() {
  const client = new ApolloClient({
    uri: "http://localhost:3000/graphql",
    cache: new InMemoryCache(),
  });
  client
    .mutate({
      mutation: CreateMoviesDocument,
      variables: {
        input: [
          {
            actors: {
              create: [
                {
                  node: {
                    name: "jacky",
                  },
                },
              ],
            },
            title: "fallout",
          },
        ],
      },
    })
    .then((res) => {
      console.log(res);
    });
}

Set up OS for developer to prevent data loss

· 2 min read
Huakun Shen
Huakun Shen
Website Owner

As a developer, I have hundreds of repos on my computer. I also need to reinstall the system every year or few months depending on the amount of garbage I added to the OS or if I broken the OS.

Data loss is a serous problem. My design goal is to prevent any possibility of losing important files even if the OS crashed, the computer is stolen, and I can reset it any time without worrying about backing up my data. (Reinstalling apps is fine for me).

The solution is simple, cloud.

My files include the following categories

  1. Code
  2. Videos
  3. Regular documents (pdf, books, forms, notes, etc.)
  4. Images

My code repos is always tracked with git and all changes must be backed up to GitHub immediately.

The code are stored in ~/Dev/

All other files are saved in OneDrive or iCloud Drive.

There is nothing on my ~/Desktop. I like to keep it clean.

This way I can reset my computer any time.

Backup​

Although I have all my files in the cloud, sometimes you may forgot to commit all changes to git. You may want to backup the entire projects folder to a drive, or cloud.

However, projects can take up a huge amount of space. My single rust project can easily take up 10-60GB due to large cache. This will take forever to upload to the cloud.

I wrote a small app called devclean (https://github.com/HuakunShen/devclean) with 2 simple feautres:

  1. Scan a given directory recursively for all git repos that have uncommitted changes.
    1. You can commit the changes and push before resetting the computer.
  2. Scan a given directory recursively for cache and dependency files (node_modules, target, etc.). These files can be deleted before backup. Then the project code is probably a few MB.
    1. Clearing cache and dependencies can save a lot of space and time during backup.

Rust SocketIO Event Handling with Channel

· 3 min read
Huakun Shen
Huakun Shen
Website Owner

rust-socketio is an open source Rust client for socket.io

It has an async version, but the event handling is a bit tricky. All callback functions (closures) have to use async move {} to handle events. To use variables outside the closure, you have to make clones of the variables and pass them to the closure. The regular sync version also needs to do this, but the async version is more complicated because of the async nature. The variables have to be cloned and moved twice.

#[tokio::main]
async fn main() {
    let (tx, rx) = channel::<String>();
    let tx2 = tx.clone();
    let socket = ClientBuilder::new("http://localhost:9559")
        .on("evt1", move |payload, socket| {
            let tx = tx.clone();
            async move {
                tx.send(format!("identity: {:?}", payload)).unwrap();
            }
            .boxed()
        })
        // .on("evt1", callback2)
        .on_any(move |evt, payload, socket| {
            let tx = tx2.clone();
            async move {
                tx.send(format!("{:?}", payload)).unwrap();
            }
            .boxed()
        })
        .connect()
        .await
        .expect("Connection failed");
}

This makes things complicated and hard to read. I have to clone variables so many times. Rust's nature keeps me focusing on the language itself rather than the business logic. In JS I could finish this without even thinking about this problem.

See this issue https://github.com/1c3t3a/rust-socketio/issues/425

Solution​

What I ended up doing is use on_any and channel to transfer all event handling to another loop outside the closures to avoid variables moving. It's much simpler.

Here is how I did it.

#[derive(Debug)]
pub struct EventMessage {
    pub event: String,
    pub payload: Payload,
}

let (done_tx, mut done_rx) = tokio::sync::mpsc::channel::<()>(1);
let (evt_tx, mut evt_rx) = tokio::sync::mpsc::channel::<EventMessage>(1);

let socket = ClientBuilder::new(SERVER_URL)
    .on(Event::Connect, |_, _| async move {}.boxed())
    .on_any(move |evt, payload, _| {
        let evt_tx = evt_tx.clone();
        async move {
            evt_tx
                .send(EventMessage {
                    event: evt.to_string(),
                    payload,
                })
                .await
                .unwrap();
        }
        .boxed()
    })
    .on(Event::Error, |err, _: Client| {
        async move {
            eprintln!("Error: {:#?}", err);
        }
        .boxed()
    })
    .connect()
    .await
    .expect("Connection failed");


loop {
    tokio::select! {
        _ = done_rx.recv() => {
            break;
        }
        Some(evt) = evt_rx.recv() => {
            // Handle event received from evt_rx
            match evt.event.as_str() {
                "evt1" => {...}
                "evt2" => {...}
            }
        }
        _ = tokio::signal::ctrl_c() => {
            break;
        }
    };
}

All events are caught by on_any and sent to evt_tx. Then I can handle all events in the loop outside the closures. This way I don't have to clone variables so many times and move them. It's much simpler and easier to read.

Not sure about the performance difference. It shouldn't matter as this is dealing with network I/O. The performance bottleneck is the network, not the CPU. So I think this is a good solution.

Cloning variables, locking and unlocking mutexes, and moving variables from stack to heap all cost time, and are harder to read. Unsure about the cost of using channels, but I think it's a good trade-off.

I am thinking about a new design for rust_socketio. The ClientBuilder can simply return a channel to the user, and the user can handle all events in the loop with select outside the closures. This way the user can handle events in a more natural way.

Magic Wormhole Source Code Analysis

· 8 min read
Huakun Shen
Huakun Shen
Website Owner

Clients​

  1. Python: https://github.com/magic-wormhole/magic-wormhole (official, original)
  2. Rust: https://github.com/magic-wormhole/magic-wormhole.rs (official)
  3. Golang: https://github.com/psanford/wormhole-william.git (non-official)
  4. Golang + Fyne GUI Client: https://github.com/Jacalz/rymdport
  5. Rust + Tauri GUI Client: https://github.com/HuakunShen/wormhole-gui

Documentation​

Performance​

magic-wormhole can almost always eat the full bandwidth of the network. It's very fast. However, I have observed performance issue on Mac (M1 pro) during sending (not receiving).

See update on issue https://github.com/magic-wormhole/magic-wormhole.rs/issues/224

Sender ComputerSender ClientReceiver ComputerReceiver ClientSpeed
M1 pro MacpythonUbuntu i7 13700Kpython112MB/s
M1 pro MacrustUbuntu i7 13700Kpython73MB/s
M1 pro MacgolangUbuntu i7 13700Kpython117MB/s
Ubuntu i7 13700KpythonM1 pro Macpython115MB/s
Ubuntu i7 13700KrustM1 pro Macpython116MB/s
Ubuntu i7 13700KgolangM1 pro Macpython117MB/s
Ubuntu i7 13700KpythonKali VM (on Mac)python119MB/s
Kali VM (on Mac)pythonUbuntu i7 13700Kpython30MB/s
Ubuntu i7 11800HrustUbuntu i7 13700Kpython116MB/s
Ubuntu i7 13700KrustUbuntu i7 11800Hpython116MB/s

It seems like there is some performance issue with the rust implementation on the sender side.

Workflow​

I read the client source code written in Python, Golang and Rust. The Python code is unreadable to me. Some packages like automat and twisted are used. I am not familiar with them and they make the code hard to read or follow. It even took me ~20-30 minutes to find the main function and get the debugger running. The code is not well-organized. It's hard to follow the workflow.

Rust is known for its complexity. It's async and await makes debugger jump everywhere. Variables allocated in heap are hard to track with debugger. Usually only a pointer address is shown.

The Golang version (although non-official) is the easiest to follow. Project structure is clear and simple. Goland's debugger works well. So let's follow the Golang version.

  • After command arguments parsing, everything starts here sendFile(args[0])

  • A wormhole.Client is created c := newClient()

  • The code is retrieved from code, status, err := c.SendFile(ctx, filepath.Base(filename), f, args...)

    • status is a channel (var status chan wormhole.SendResult) that waits for the result of sending file.
    •     s := <-status

          if s.OK {
              fmt.Println("file sent")
          } else {
              bail("Send error: %s", s.Error)
          }

  • Here is Wormhole Client's SendFile() method

    •     func (c *Client) SendFile(ctx context.Context, fileName string, r io.ReadSeeker, opts ...SendOption) (string, chan SendResult, error) {
              if err := c.validateRelayAddr(); err != nil {
                  return "", nil, fmt.Errorf("invalid TransitRelayAddress: %s", err)
              }

              size, err := readSeekerSize(r)
              if err != nil {
                  return "", nil, err
              }

              offer := &offerMsg{
                  File: &offerFile{
                      FileName: fileName,
                      FileSize: size,
                  },
              }

              return c.sendFileDirectory(ctx, offer, r, opts...)
          }

    • offer contains the file name and size.

  • Let's go into sendFileDirectory() method here. Everything happens here.

    • sideId: RandSideID returns a string appropate for use as the Side ID for a client.

      NewClient returns a Rendezvous client. URL is the websocket url of Rendezvous server. SideID is the id for the client to use to distinguish messages in a mailbox from the other client. AppID is the application identity string of the client.

      Two clients can only communicate if they have the same AppID.

      sideID := crypto.RandSideID()
      appID := c.appID()
      rc := rendezvous.NewClient(c.url(), sideID, appID)

      _, err := rc.Connect(ctx)

    • Then a nameplate is generated

      If users provides the code, the mailbox is attached to the code. Otherwise, a new mailbox is created. A mailbox is a channel for communication between two clients. The sender creates a mailbox and sends the code (address of mailbox + key) to the receiver. The receiver uses the code to open the mailbox.

      if options.code == "" {
        // CreateMailbox allocates a nameplate, claims it, and then opens the associated mailbox. It returns the nameplate id string.
        // nameplate is a number string. e.g. 10
        nameplate, err := rc.CreateMailbox(ctx)
        if err != nil {
          return "", nil, err
        }

        // ChooseWords returns 2 words from the wordlist. (e.g. "correct-horse")
        pwStr = nameplate + "-" + wordlist.ChooseWords(c.wordCount())
      } else {
        pwStr = options.code
        nameplate, err := nameplateFromCode(pwStr)
        if err != nil {
          return "", nil, err
        }

        // AttachMailbox opens an existing mailbox and releases the associated nameplate.
        err = rc.AttachMailbox(ctx, nameplate)
        if err != nil {
          return "", nil, err
        }
      }

    • Then a clientProto is created

        clientProto := newClientProtocol(ctx, rc, sideID, appID)

      appID is a constant string. sideID is a random string.

      sideID := crypto.RandSideID() RandSideID returns a string appropate for use as the Side ID for a client.

      Let's see how newClientProtocol works.

        type clientProtocol struct {
          sharedKey    []byte
          phaseCounter int
          ch           <-chan rendezvous.MailboxEvent
          rc           *rendezvous.Client
          spake        *gospake2.SPAKE2
          sideID       string
          appID        string
        }

        func newClientProtocol(ctx context.Context, rc *rendezvous.Client, sideID, appID string) *clientProtocol {
          recvChan := rc.MsgChan(ctx)

          return &clientProtocol{
            ch:     recvChan,
            rc:     rc,
            sideID: sideID,
            appID:  appID,
          }
        }

    • Then enter a go routing (transfer happens here)

      • clinetProto.ReadPake(ctx): block and waiting for receiver to connect (wait for receiver to enter the code)

        ReadPake calls readPlainText to read the event from the mailbox.

        func (cc *clientProtocol) readPlaintext(ctx context.Context, phase string, v interface{}) error {
          var gotMsg rendezvous.MailboxEvent
          select {
          case gotMsg = <-cc.ch:
          case <-ctx.Done():
            return ctx.Err()
          }
          if gotMsg.Error != nil {
            return gotMsg.Error
          }

          if gotMsg.Phase != phase {
            return fmt.Errorf("got unexpected phase while waiting for %s: %s", phase, gotMsg.Phase)
          }

          err := jsonHexUnmarshal(gotMsg.Body, &v)
          if err != nil {
            return err
          }

          return nil
        }

        func (cc *clientProtocol) ReadPake(ctx context.Context) error {
          var pake pakeMsg
          err := cc.readPlaintext(ctx, "pake", &pake)
          if err != nil {
            return err
          }

          otherSidesMsg, err := hex.DecodeString(pake.Body)
          if err != nil {
            return err
          }

          sharedKey, err := cc.spake.Finish(otherSidesMsg)
          if err != nil {
            return err
          }

          cc.sharedKey = sharedKey

          return nil
        }

        pake's body is a string of length 66. otherSidesMsg is []uint8 bytes of length 33.

        Then sharedKey is generated by calling cc.spake.Finish(otherSidesMsg). spake is a SPAKE2 object.

        sharedKey is a 32-byte long byte array.

        So what is pake message read from the mailbox?

        TODO

      • err = collector.waitFor(&answer): Wait for receiver to enter Y to confirm. The answer contains a OK message

      • A cryptor (type=transportCryptor) is created.

          cryptor := newTransportCryptor(conn, transitKey, "transit_record_receiver_key", "transit_record_sender_key")

          recordSize := (1 << 14) // record size: 16384 byte (16kb)
          // chunk
          recordSlice := make([]byte, recordSize-secretbox.Overhead)
          hasher := sha256.New()

        conn is a net.TCPConn TCP connection.

        A readKey and writeKey are generated with hkdf (HMAC-based Extract-and-Expand Key Derivation Function) from transitKey and two strings in newTransportCryptor.

        transitKey is derived from clientProto.sharedKey and appID.

        transitKey := deriveTransitKey(clientProto.sharedKey, appID)

        sharedKey is a 32-byte long key generated by clientProto (a pake.Client).

        func newTransportCryptor(c net.Conn, transitKey []byte, readPurpose, writePurpose string) *transportCryptor {
          r := hkdf.New(sha256.New, transitKey, nil, []byte(readPurpose))
          var readKey [32]byte
          _, err := io.ReadFull(r, readKey[:])
          if err != nil {
            panic(err)
          }

          r = hkdf.New(sha256.New, transitKey, nil, []byte(writePurpose))
          var writeKey [32]byte
          _, err = io.ReadFull(r, writeKey[:])
          if err != nil {
            panic(err)
          }

          return &transportCryptor{
            conn:          c,
            prefixBuf:     make([]byte, 4+crypto.NonceSize),
            nextReadNonce: big.NewInt(0),
            readKey:       readKey,
            writeKey:      writeKey,
          }
        }

        recordSize is 16384 byte (16kb), used to read file in chunks.

        hasher is compute file hash while reading file.

      • In the following loop, file is read and sent in chunks.

        r has type io.Reader. Every time 16KB is read.

        cryptor.writeRecord encrypts the bytes and send the bytes.

        for {
          n, err := r.Read(recordSlice)
          if n > 0 {
            hasher.Write(recordSlice[:n])
            err = cryptor.writeRecord(recordSlice[:n]) // send 16KB in each iteration
            if err != nil {
              sendErr(err)
              return
            }
            progress += int64(n)
            if options.progressFunc != nil {
              options.progressFunc(progress, totalSize)
            }
          }
          if err == io.EOF {
            break
          } else if err != nil {
            sendErr(err)
            return
          }
        }

        Let's see how writeRecord works.

        package secretbox ("golang.org/x/crypto/nacl/secretbox") is used to encrypt data.

        d.conn.Write sends the encrypted data out.

        func (d *transportCryptor) writeRecord(msg []byte) error {
          var nonce [crypto.NonceSize]byte

          if d.nextWriteNonce == math.MaxUint64 {
            panic("Nonce exhaustion")
          }

          binary.BigEndian.PutUint64(nonce[crypto.NonceSize-8:], d.nextWriteNonce)
          d.nextWriteNonce++

          sealedMsg := secretbox.Seal(nil, msg, &nonce, &d.writeKey)

          nonceAndSealedMsg := append(nonce[:], sealedMsg...)

          // we do an explit cast to int64 to avoid compilation failures
          // for 32bit systems.
          nonceAndSealedMsgSize := int64(len(nonceAndSealedMsg))

          if nonceAndSealedMsgSize >= math.MaxUint32 {
            panic(fmt.Sprintf("writeRecord too large: %d", len(nonceAndSealedMsg)))
          }

          l := make([]byte, 4)
          binary.BigEndian.PutUint32(l, uint32(len(nonceAndSealedMsg)))

          lenNonceAndSealedMsg := append(l, nonceAndSealedMsg...)

          _, err := d.conn.Write(lenNonceAndSealedMsg)
          return err
        }

Cloud Run Mount Volume Secret

· 2 min read
Huakun Shen
Huakun Shen
Website Owner

Cloud Run is a good serverless solution for deploying docker containers. It's easy to deploy and scale. Normally, if you can run a docker container in local dev environment, you can deploy it to cloud run directly without too much extra configuration.

However, in my experience, there is one thing that is easy to waste lots of time on, that is mounting a secret file to the container.

Volumes​

Cloud run allows you to add environment variables, but one by one. It's not convenient if you have a lot of environment variables, and if you need to change them often.

My solution is to add the content of .env file to Secret Manager in GCP and mount the secret file to the container, then load the .env file in the source code. This way I can update all env vars at once by creating a new version of the secret.

With docker volum, we can mount a single file easily like this docker run -v ./secret.env:/app/.env image-name.

However in cloud run, it's not that easy. If you try to configure the volume the same way docker does, your container will fail to start.

Here is the process to mount a secret file to cloud run;

  • Under volumes tab, you can add a secret volume type, choose a secret from Secret Manager.
  • The mount path can be a filename, such as .env.
  • Then go to the Container(s) tab. Under VOLUME MOUNTS you can add Volume Mount.

The mount path points to the folder where the secret file is mounted, but the folder has to be empty/non-existent in your source code. Cloud Run doesn't allow mounting a single file, the mounted folder will replace the folder in your source code, because the folder is a protected folder by GCP.

If your source code is in /xc-server, and the mount path is set to /xc-server with the mounted file at /xc-server/.env, then the /xc-server folder will be completely removed and contain only the .env file.

What I do is mount the folder to /xc-server/env/.env, then in the source code load the .env file from /xc-server/env/.env.

Docker ca-certificates Dependency Required by Prisma

· One min read
Huakun Shen
Huakun Shen
Website Owner

When I tried to run prisma within a docker container, I got this error:

Error opening a TLS connection: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:../ssl/statem/statem_clnt.c:1919: (unable to get local issuer certificate)

The docker image I used was oven/bun, but I believe this error can happen to any docker image that doesn't install ca-certificates.

The solution is simple, add the following to the Dockerfile

RUN apt update && apt install -y ca-certificates

What is ca-certificates​

https://packages.debian.org/sid/ca-certificates

Contains the certificate authorities shipped with Mozilla's browser to allow SSL-based applications to check for the authenticity of SSL connections.

Browsers like chrome or firefox have built-in trusted certificate authorities, which means they can communicate to verify the authenticity of SSL connections. When your prisma connect requires SSL connection, then you have to install ca-certificates to allow SSL-based applications to check for the authenticity of SSL connections.