As a developer, I have hundreds of repos on my computer. I also need to reinstall the system every year or few months depending on the amount of garbage I added to the OS or if I broken the OS.

Data loss is a serous problem. My design goal is to prevent any possibility of losing important files even if the OS crashed, the computer is stolen, and I can reset it any time without worrying about backing up my data. (Reinstalling apps is fine for me).

The solution is simple, cloud.

My files include the following categories

1. Code
2. Videos
3. Regular documents (pdf, books, forms, notes, etc.)
4. Images

My code repos is always tracked with git and all changes must be backed up to GitHub immediately.

The code are stored in ~/Dev/

All other files are saved in OneDrive or iCloud Drive.

There is nothing on my ~/Desktop. I like to keep it clean.

This way I can reset my computer any time.

Backup​

Although I have all my files in the cloud, sometimes you may forgot to commit all changes to git. You may want to backup the entire projects folder to a drive, or cloud.

However, projects can take up a huge amount of space. My single rust project can easily take up 10-60GB due to large cache. This will take forever to upload to the cloud.

I wrote a small app called devclean (https://github.com/HuakunShen/devclean) with 2 simple feautres:

1. Scan a given directory recursively for all git repos that have uncommitted changes.
   1. You can commit the changes and push before resetting the computer.
2. Scan a given directory recursively for cache and dependency files (node_modules, target, etc.). These files can be deleted before backup. Then the project code is probably a few MB.
   1. Clearing cache and dependencies can save a lot of space and time during backup.

rust-socketio is an open source Rust client for socket.io

It has an async version, but the event handling is a bit tricky. All callback functions (closures) have to use async move {} to handle events. To use variables outside the closure, you have to make clones of the variables and pass them to the closure. The regular sync version also needs to do this, but the async version is more complicated because of the async nature. The variables have to be cloned and moved twice.

#[tokio::main]
async fn main() {
    let (tx, rx) = channel::<String>();
    let tx2 = tx.clone();
    let socket = ClientBuilder::new("http://localhost:9559")
        .on("evt1", move |payload, socket| {
            let tx = tx.clone();
            async move {
                tx.send(format!("identity: {:?}", payload)).unwrap();
            }
            .boxed()
        })
        // .on("evt1", callback2)
        .on_any(move |evt, payload, socket| {
            let tx = tx2.clone();
            async move {
                tx.send(format!("{:?}", payload)).unwrap();
            }
            .boxed()
        })
        .connect()
        .await
        .expect("Connection failed");
}

This makes things complicated and hard to read. I have to clone variables so many times. Rust's nature keeps me focusing on the language itself rather than the business logic. In JS I could finish this without even thinking about this problem.

See this issue https://github.com/1c3t3a/rust-socketio/issues/425

Solution​

What I ended up doing is use on_any and channel to transfer all event handling to another loop outside the closures to avoid variables moving. It's much simpler.

Here is how I did it.

#[derive(Debug)]
pub struct EventMessage {
    pub event: String,
    pub payload: Payload,
}

let (done_tx, mut done_rx) = tokio::sync::mpsc::channel::<()>(1);
let (evt_tx, mut evt_rx) = tokio::sync::mpsc::channel::<EventMessage>(1);

let socket = ClientBuilder::new(SERVER_URL)
    .on(Event::Connect, |_, _| async move {}.boxed())
    .on_any(move |evt, payload, _| {
        let evt_tx = evt_tx.clone();
        async move {
            evt_tx
                .send(EventMessage {
                    event: evt.to_string(),
                    payload,
                })
                .await
                .unwrap();
        }
        .boxed()
    })
    .on(Event::Error, |err, _: Client| {
        async move {
            eprintln!("Error: {:#?}", err);
        }
        .boxed()
    })
    .connect()
    .await
    .expect("Connection failed");


loop {
    tokio::select! {
        _ = done_rx.recv() => {
            break;
        }
        Some(evt) = evt_rx.recv() => {
            // Handle event received from evt_rx
            match evt.event.as_str() {
                "evt1" => {...}
                "evt2" => {...}
            }
        }
        _ = tokio::signal::ctrl_c() => {
            break;
        }
    };
}

All events are caught by on_any and sent to evt_tx. Then I can handle all events in the loop outside the closures. This way I don't have to clone variables so many times and move them. It's much simpler and easier to read.

Not sure about the performance difference. It shouldn't matter as this is dealing with network I/O. The performance bottleneck is the network, not the CPU. So I think this is a good solution.

Cloning variables, locking and unlocking mutexes, and moving variables from stack to heap all cost time, and are harder to read. Unsure about the cost of using channels, but I think it's a good trade-off.

I am thinking about a new design for rust_socketio. The ClientBuilder can simply return a channel to the user, and the user can handle all events in the loop with select outside the closures. This way the user can handle events in a more natural way.

Clients​

1. Python: https://github.com/magic-wormhole/magic-wormhole (official, original)
2. Rust: https://github.com/magic-wormhole/magic-wormhole.rs (official)
3. Golang: https://github.com/psanford/wormhole-william.git (non-official)
4. Golang + Fyne GUI Client: https://github.com/Jacalz/rymdport
5. Rust + Tauri GUI Client: https://github.com/HuakunShen/wormhole-gui

Documentation​

• https://github.com/magic-wormhole/magic-wormhole-protocols
• https://magic-wormhole.readthedocs.io/en/latest/

Performance​

magic-wormhole can almost always eat the full bandwidth of the network. It's very fast. However, I have observed performance issue on Mac (M1 pro) during sending (not receiving).

See update on issue https://github.com/magic-wormhole/magic-wormhole.rs/issues/224

It seems like there is some performance issue with the rust implementation on the sender side.

Workflow​

I read the client source code written in Python, Golang and Rust. The Python code is unreadable to me. Some packages like automat and twisted are used. I am not familiar with them and they make the code hard to read or follow. It even took me ~20-30 minutes to find the main function and get the debugger running. The code is not well-organized. It's hard to follow the workflow.

Rust is known for its complexity. It's async and await makes debugger jump everywhere. Variables allocated in heap are hard to track with debugger. Usually only a pointer address is shown.

The Golang version (although non-official) is the easiest to follow. Project structure is clear and simple. Goland's debugger works well. So let's follow the Golang version.

• After command arguments parsing, everything starts here sendFile(args[0])

• A wormhole.Client is created c := newClient()

• The code is retrieved from code, status, err := c.SendFile(ctx, filepath.Base(filename), f, args...)

  • status is a channel (var status chan wormhole.SendResult) that waits for the result of sending file.

  •     s := <-status
    
        if s.OK {
            fmt.Println("file sent")
        } else {
            bail("Send error: %s", s.Error)
        }
    
    

• Here is Wormhole Client's SendFile() method

  •     func (c *Client) SendFile(ctx context.Context, fileName string, r io.ReadSeeker, opts ...SendOption) (string, chan SendResult, error) {
            if err := c.validateRelayAddr(); err != nil {
                return "", nil, fmt.Errorf("invalid TransitRelayAddress: %s", err)
            }
    
            size, err := readSeekerSize(r)
            if err != nil {
                return "", nil, err
            }
    
            offer := &offerMsg{
                File: &offerFile{
                    FileName: fileName,
                    FileSize: size,
                },
            }
    
            return c.sendFileDirectory(ctx, offer, r, opts...)
        }
    

  • offer contains the file name and size.

• Let's go into sendFileDirectory() method here. Everything happens here.

  • sideId: RandSideID returns a string appropate for use as the Side ID for a client.

    NewClient returns a Rendezvous client. URL is the websocket url of Rendezvous server. SideID is the id for the client to use to distinguish messages in a mailbox from the other client. AppID is the application identity string of the client.

    Two clients can only communicate if they have the same AppID.

    sideID := crypto.RandSideID()
    appID := c.appID()
    rc := rendezvous.NewClient(c.url(), sideID, appID)
    
    _, err := rc.Connect(ctx)
    

  • Then a nameplate is generated

    If users provides the code, the mailbox is attached to the code. Otherwise, a new mailbox is created. A mailbox is a channel for communication between two clients. The sender creates a mailbox and sends the code (address of mailbox + key) to the receiver. The receiver uses the code to open the mailbox.

    if options.code == "" {
      // CreateMailbox allocates a nameplate, claims it, and then opens the associated mailbox. It returns the nameplate id string.
      // nameplate is a number string. e.g. 10
      nameplate, err := rc.CreateMailbox(ctx)
      if err != nil {
        return "", nil, err
      }
    
      // ChooseWords returns 2 words from the wordlist. (e.g. "correct-horse")
      pwStr = nameplate + "-" + wordlist.ChooseWords(c.wordCount())
    } else {
      pwStr = options.code
      nameplate, err := nameplateFromCode(pwStr)
      if err != nil {
        return "", nil, err
      }
    
      // AttachMailbox opens an existing mailbox and releases the associated nameplate.
      err = rc.AttachMailbox(ctx, nameplate)
      if err != nil {
        return "", nil, err
      }
    }
    

  • Then a clientProto is created

      clientProto := newClientProtocol(ctx, rc, sideID, appID)
    

    appID is a constant string. sideID is a random string.

    sideID := crypto.RandSideID() RandSideID returns a string appropate for use as the Side ID for a client.

    Let's see how newClientProtocol works.

      type clientProtocol struct {
        sharedKey    []byte
        phaseCounter int
        ch           <-chan rendezvous.MailboxEvent
        rc           *rendezvous.Client
        spake        *gospake2.SPAKE2
        sideID       string
        appID        string
      }
    
      func newClientProtocol(ctx context.Context, rc *rendezvous.Client, sideID, appID string) *clientProtocol {
        recvChan := rc.MsgChan(ctx)
    
        return &clientProtocol{
          ch:     recvChan,
          rc:     rc,
          sideID: sideID,
          appID:  appID,
        }
      }
    

  • Then enter a go routing (transfer happens here)

    • clinetProto.ReadPake(ctx): block and waiting for receiver to connect (wait for receiver to enter the code)

      ReadPake calls readPlainText to read the event from the mailbox.

      func (cc *clientProtocol) readPlaintext(ctx context.Context, phase string, v interface{}) error {
        var gotMsg rendezvous.MailboxEvent
        select {
        case gotMsg = <-cc.ch:
        case <-ctx.Done():
          return ctx.Err()
        }
        if gotMsg.Error != nil {
          return gotMsg.Error
        }
      
        if gotMsg.Phase != phase {
          return fmt.Errorf("got unexpected phase while waiting for %s: %s", phase, gotMsg.Phase)
        }
      
        err := jsonHexUnmarshal(gotMsg.Body, &v)
        if err != nil {
          return err
        }
      
        return nil
      }
      
      func (cc *clientProtocol) ReadPake(ctx context.Context) error {
        var pake pakeMsg
        err := cc.readPlaintext(ctx, "pake", &pake)
        if err != nil {
          return err
        }
      
        otherSidesMsg, err := hex.DecodeString(pake.Body)
        if err != nil {
          return err
        }
      
        sharedKey, err := cc.spake.Finish(otherSidesMsg)
        if err != nil {
          return err
        }
      
        cc.sharedKey = sharedKey
      
        return nil
      }
      
      

      pake's body is a string of length 66. otherSidesMsg is []uint8 bytes of length 33.

      Then sharedKey is generated by calling cc.spake.Finish(otherSidesMsg). spake is a SPAKE2 object.

      sharedKey is a 32-byte long byte array.

      So what is pake message read from the mailbox?

      TODO

    • err = collector.waitFor(&answer): Wait for receiver to enter Y to confirm. The answer contains a OK message

    • A cryptor (type=transportCryptor) is created.

        cryptor := newTransportCryptor(conn, transitKey, "transit_record_receiver_key", "transit_record_sender_key")
      
        recordSize := (1 << 14) // record size: 16384 byte (16kb)
        // chunk
        recordSlice := make([]byte, recordSize-secretbox.Overhead)
        hasher := sha256.New()
      

      conn is a net.TCPConn TCP connection.

      A readKey and writeKey are generated with hkdf (HMAC-based Extract-and-Expand Key Derivation Function) from transitKey and two strings in newTransportCryptor.

      transitKey is derived from clientProto.sharedKey and appID.

      transitKey := deriveTransitKey(clientProto.sharedKey, appID)
      

      sharedKey is a 32-byte long key generated by clientProto (a pake.Client).

      func newTransportCryptor(c net.Conn, transitKey []byte, readPurpose, writePurpose string) *transportCryptor {
        r := hkdf.New(sha256.New, transitKey, nil, []byte(readPurpose))
        var readKey [32]byte
        _, err := io.ReadFull(r, readKey[:])
        if err != nil {
          panic(err)
        }
      
        r = hkdf.New(sha256.New, transitKey, nil, []byte(writePurpose))
        var writeKey [32]byte
        _, err = io.ReadFull(r, writeKey[:])
        if err != nil {
          panic(err)
        }
      
        return &transportCryptor{
          conn:          c,
          prefixBuf:     make([]byte, 4+crypto.NonceSize),
          nextReadNonce: big.NewInt(0),
          readKey:       readKey,
          writeKey:      writeKey,
        }
      }
      

      recordSize is 16384 byte (16kb), used to read file in chunks.

      hasher is compute file hash while reading file.

    • In the following loop, file is read and sent in chunks.

      r has type io.Reader. Every time 16KB is read.

      cryptor.writeRecord encrypts the bytes and send the bytes.

      for {
        n, err := r.Read(recordSlice)
        if n > 0 {
          hasher.Write(recordSlice[:n])
          err = cryptor.writeRecord(recordSlice[:n]) // send 16KB in each iteration
          if err != nil {
            sendErr(err)
            return
          }
          progress += int64(n)
          if options.progressFunc != nil {
            options.progressFunc(progress, totalSize)
          }
        }
        if err == io.EOF {
          break
        } else if err != nil {
          sendErr(err)
          return
        }
      }
      

      Let's see how writeRecord works.

      package secretbox ("golang.org/x/crypto/nacl/secretbox") is used to encrypt data.

      d.conn.Write sends the encrypted data out.

      func (d *transportCryptor) writeRecord(msg []byte) error {
        var nonce [crypto.NonceSize]byte
      
        if d.nextWriteNonce == math.MaxUint64 {
          panic("Nonce exhaustion")
        }
      
        binary.BigEndian.PutUint64(nonce[crypto.NonceSize-8:], d.nextWriteNonce)
        d.nextWriteNonce++
      
        sealedMsg := secretbox.Seal(nil, msg, &nonce, &d.writeKey)
      
        nonceAndSealedMsg := append(nonce[:], sealedMsg...)
      
        // we do an explit cast to int64 to avoid compilation failures
        // for 32bit systems.
        nonceAndSealedMsgSize := int64(len(nonceAndSealedMsg))
      
        if nonceAndSealedMsgSize >= math.MaxUint32 {
          panic(fmt.Sprintf("writeRecord too large: %d", len(nonceAndSealedMsg)))
        }
      
        l := make([]byte, 4)
        binary.BigEndian.PutUint32(l, uint32(len(nonceAndSealedMsg)))
      
        lenNonceAndSealedMsg := append(l, nonceAndSealedMsg...)
      
        _, err := d.conn.Write(lenNonceAndSealedMsg)
        return err
      }
      

Cloud Run is a good serverless solution for deploying docker containers. It's easy to deploy and scale. Normally, if you can run a docker container in local dev environment, you can deploy it to cloud run directly without too much extra configuration.

However, in my experience, there is one thing that is easy to waste lots of time on, that is mounting a secret file to the container.

Volumes​

Cloud run allows you to add environment variables, but one by one. It's not convenient if you have a lot of environment variables, and if you need to change them often.

My solution is to add the content of .env file to Secret Manager in GCP and mount the secret file to the container, then load the .env file in the source code. This way I can update all env vars at once by creating a new version of the secret.

With docker volum, we can mount a single file easily like this docker run -v ./secret.env:/app/.env image-name.

However in cloud run, it's not that easy. If you try to configure the volume the same way docker does, your container will fail to start.

Here is the process to mount a secret file to cloud run;

• Under volumes tab, you can add a secret volume type, choose a secret from Secret Manager.
• The mount path can be a filename, such as .env.
• Then go to the Container(s) tab. Under VOLUME MOUNTS you can add Volume Mount.

The mount path points to the folder where the secret file is mounted, but the folder has to be empty/non-existent in your source code. Cloud Run doesn't allow mounting a single file, the mounted folder will replace the folder in your source code, because the folder is a protected folder by GCP.

If your source code is in /xc-server, and the mount path is set to /xc-server with the mounted file at /xc-server/.env, then the /xc-server folder will be completely removed and contain only the .env file.

What I do is mount the folder to /xc-server/env/.env, then in the source code load the .env file from /xc-server/env/.env.

When I tried to run prisma within a docker container, I got this error:

Error opening a TLS connection: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:../ssl/statem/statem_clnt.c:1919: (unable to get local issuer certificate)

The docker image I used was oven/bun, but I believe this error can happen to any docker image that doesn't install ca-certificates.

The solution is simple, add the following to the Dockerfile

RUN apt update && apt install -y ca-certificates

What is ca-certificates​

https://packages.debian.org/sid/ca-certificates

Contains the certificate authorities shipped with Mozilla's browser to allow SSL-based applications to check for the authenticity of SSL connections.

Browsers like chrome or firefox have built-in trusted certificate authorities, which means they can communicate to verify the authenticity of SSL connections. When your prisma connect requires SSL connection, then you have to install ca-certificates to allow SSL-based applications to check for the authenticity of SSL connections.

My project CrossCopy has been using the following tech stack

• npm for package management
• nodejs for JS runtime
• Server: expressjs + nodejs
• Web: Svelte + nodejs
• Monorepo: nx monorepo + turborepo + npm workspaces
• JS Module: CJS
• API: GraphQL
• Sync: GraphQL subscription

Recently I spend a few days migrating from this stack to the following stack

• pnpm + bun for package management
• bun for JS runtime
• Server: Nestjs + bun
• Web: Svelte + bun
• Monorepo: turborepo + (pnpm workspaces + bun workspaces)
  • bun and pnpm workspaces are pretty much the same
• JS Module: ESM
• API: GraphQL
• Sync: SocketIO
  • Bun currently doesn't work with Apollo GraphQL's subscription. https://github.com/oven-sh/bun/issues/6905

This makes the project much easier to develop and maintain. This is a blog I copied from our dev docs and may be useful for people who are interested in using bun for their projects.

There are huge breaking changes in this migration to improve development experience and performance.

This blog is must read, there are many places to be aware of, otherwise you may not be able to run the project.

Here are the PRs in our two main monorepo repositories for server and clients:

• https://github.com/CrossCopy/crosscopy-clients/pull/16
• https://github.com/CrossCopy/crosscopy-dev/pull/98

Many changes was maade in the @crosscopy/core and @crosscopy/graphql-schema repo but they don't have a separate PR for simplicity. Check the associated commits in the 2 PRs above if you really want.

After tons of research and experiment, I decided to make this huge refactoring and toolchain migration.

• Migrate nodejs runtime to bun runtime (https://bun.sh/)
  • Bun is much mcuh faster than nodejs
  • For now I removed the subpath exports in package.json for @crosscopy/core and @crosscopy/graphql which requires building TypeScript into JavaScript first, and then import them in another project. This is complicated during development, especially the 2 libraries are only used by ourselves. Bun allows us to import TypeScript directly using relative paths, we no longer need to build them. Before, after making any changes in @crosscopy/core, I have to manually compile it again before I can see the changes in server who imports it. Now, I can see the changes immediately. Dev servers with bun --watch will pick up the changes from it's dependencies automatically although it's in another package, there is no need to restart server, without feeling another separate package.
  • Bun runs TypeScript directly, no need to build TypeScript into JavaScript first, which is much faster.
  • Environment Variables
    • With nodejs and ts-node, we have to use a dotenv package to programmatically load environment variables from .env file.
    • Bun has a built-in support for .env file, we no longer need to use dotenv package. See https://bun.sh/guides/runtime/set-env and https://bun.sh/guides/runtime/read-env
    • When running and command with bun, bun run <script> or bunx vitest, bun will automatically load .env file in the current directory, making testing and development easier. This is why the test script in package.json for many packages are changed to things like bunx vitest or bun run test. Sometimes npm run test won't work because it doesn't load .env file.
• Migrate package manager to pnpm
  • bun is not only a runtime, but also a package manager. It's the fastest I've ever seen. Faster than npm, yarn and pnpm. During migration to bun runtime I always use bun to install packages and do package management. I changed my plan when I started migrating clients CICD to use bun, as bun currently only works on MacOS and Linux, not Windows. Our future Windows client will have to be built on Windows in CICD, and our development environment should support Windows although I personally use MacOS and Linux all the time.
  • Using npm to do package management is no longer possible because bun workspaces uses slightly different syntax from npm workspaces.
    • Using packages from the same monorepo as dependency requires adding the package name to package.json. npm workspaces uses "@crosscopy/core": "*" syntax, while bun workspaces uses "@crosscopy/core": "workspace:*". The extra workspace: prefix is required and prevent npm workspaces to work with bun workspaces. i.e. npm package management and bun runtime simply won't work together in the same mono project.
  • pnpm comes to resecure. pnpm ranks first in 2022 stateofjs for monorepo tools (https://2022.stateofjs.com/en-US/libraries/monorepo-tools/). Turbo repo is second and we are using them together for our monorepo management. pnpm workspaces uses the same syntax as bun workspaces with a workspace:* prefix for within-monorepo dependencies. Making it possible to use pnpm for package management and bun for runtime in the same monorepo. Perfect! Problem solved.
• Migration to ESM from CJS.
  • CommonJS was a legacy module system of the JavaScript ecosystem. We used to use CJS for compatibility with most packages. Some packages migrate to only ESM recently, causing problems. ESM has more features and is easier to use, for example top-level await is only in ESM.
  • We now migrate every package to ESM except for the new server written with Nest.js, I will talk more about it.
• Migrate express to nestjs (https://nestjs.com/)
  • We used to use Express.js as our backend server framework. It's popular and undoubtedly the most popular framework in JS. However, it's not designed for large projects. It's hard to maintain and scale. Nest.js is a framework designed for large projects. It's based on Express.js, but it's much more powerful. It's also written in TypeScript, which is a big plus.
  • Express.js is a barebone framework, with no template, developers have to design and write everything from scratch. This is great, but bad initial design could lead to unmaintainable code in the future.
    • Our previous version server worked fine, but after the project get's bigger and a few refactor, I realized that the initial design was not good enough for this project as it grows bigger and bigger.
  • Nest.js has a lot of built-in features and templates. It organizes everything in the OOP way just like Spring Boot, many many files, but easier to read and maintain. With lots of built-in features that work out of the box, like Auth with JWT, rate limit throttler, GraphQL, Websocket with SocketIO, middleware and interceptor and much more. I don't need to set up everything from scratch, connecting components manually and making the code ugly and hard to maintain.
    • The testing framework is more mature, and it's easier to write tests. Everything is separated into modules, and it's easier to mock dependencies.
  • Problem with Nest.js.
    • I rewrite the server in Nest.js, fantastic experience and I can expect a better development and testing experience with it. However, a limitation of Nestjs is that it's completely in CommonJS, not possible to migrate to ESM. Our dependency packages (core and graphql-schema) has been migrated to the newer ESM standard, and to work with nest, they have to be compiled to CJS JavaScript first before they can be imported into Nest.js server, which gives up the freedom of importing TypeScript freely from any location.
    • Another problem with Nest.js + bun is that GraphQL subscription doesn't work with bun.
      • This is not a problem with Nest.js actually, but a problem with bun + Apollo server. bun's developer has worked so hard to make bun seamlessly integrate as a drop-in replacement for nodejs runtime by implementing most of the nodejs APIs. Most of the time I can use bun as a drop-in replacement for nodejs runtime. Bun works with nest.js websocket, but not with Apollo Server subscription. I don't know the reason either, but probably due to some missing APIs, there is no error shown. After hours of debugging, I found that bun simply won't work with Apollo Server, even without Nest.js. So it's not a Nest.js problem, but a problem between bun and Apollo Server.
      • Luckily, GraphQL Query and Mutation still work with bun runtime as they are simply HTTP requests under the hood. And since we have already decided to use SocketIO for realtime synchronization as it's more flexible and powerful than GraphQL subscription (SocketIO is two-way while subscription is only one-way), we don't need to use GraphQL subscription anymore. So this is not a problem for us. Later if Bun supports apollo server, we can use subscription again for some other simpler use cases that doesn't require two-way communication.
• Migrate crosscopy-dev repo from using nx monorepo to turborepo. I simply had more bugs and issues with nx repo. crosscopy-clients repo uses turborepo and has a better experience, so I decided to migrate crosscopy-dev repo to turborepo as well. turborepo also ranks higher than nx in 2022 stateofjs for monorepo tools (https://2022.stateofjs.com/en-US/libraries/monorepo-tools/), with 10% more retention rate, 14% more interest.

Note​

• Install bun and pnpm. bun work similar to nodejs, pnpm works similar to npm.
• In most packages, bun dev is used to start development server, bun run test is used to run tests, bun run build is used to build TypeScript into JavaScript. bun run is used to run any script in package.json.
• pnpm run build and pnpm run test in the root of a mono repo will use turborepo to build all packages in the monorepo. I've configured everything to work. If you need to run tests in subpackages, try to use bun run test or bunx vitest or bunx jest as I didn't write code to load .env file, using a bun command does that for us even if the test still uses nodejs under the hood. As long as bun is used as the initial command, .env. is loaded.
• If you are unsure about the command to use, look at .github/workflows to see what commands CI uses to build the repo. If CI works, then so should your local environment work if configured correctly.

Here is a Drag and Drop wrapper component I wrote for svelte. It's important to note that, in order for the drop event to be fired properly, we also need to call preentDefault() on the dragover event.

<script lang="ts">
	import { createEventDispatcher } from 'svelte';

	const dispatch = createEventDispatcher();

	function onDrop(e: DragEvent) {
		e.preventDefault();
		e.stopPropagation();
		dispatch('drop', e);
	}

	function onDragOver(e: DragEvent) {
		e.preventDefault();
		dispatch('dragover', e);
	}
</script>

<span
	class={$$props.class}
	on:dragenter
	on:drop={onDrop}
	on:dragleave
	on:dragover={onDragOver}
	on:dragend
	on:click
	role="none"
>
	<slot />
</span>

To use the component, here is an example. The UI requires tailwind and skeleton library to properly render. Just read the code to get the idea, it's quite simple to use in fact. DragNDrop is a wrapper, you need to set the size, and styles for it, then it's region will be file-droppable.

<script lang="ts">
	import DragNDrop from './drag-n-drop.svelte';

	function onDrop(e: CustomEvent<DragEvent>) {
		console.log(e.detail.dataTransfer?.files);
	}

	let isOver = false;
	let fileInputEle: HTMLInputElement;
</script>

<DragNDrop
	class="flex flex-col justify-center items-center space-y-6 card {$$props.class} cursor-pointer"
	on:click={() => {
		fileInputEle.click();
	}}
	on:drop={onDrop}
	on:dragleave={() => {
		isOver = false;
	}}
	on:dragenter={() => {
		isOver = true;
	}}
>
	<iconify-icon class="text-6xl" icon="ic:round-upload" />
	{#if isOver}
		<span class="text-lg"><strong>Drop the file</strong></span>
	{:else}
		<span class="text-lg"><strong>Upload a file</strong> or drag and drop</span>
	{/if}
	<input
		type="file"
		class="hidden"
		multiple
		bind:this={fileInputEle}
		on:change={(e) => {
			// console.log(e.target);
		}}
	/>
</DragNDrop>

Let's say you have a local network with subnet 192.168.0.0/24, the router is on 192.168.0.1, you have a VM running on 192.168.0.2 (Doesn't matter here).

I want to give a teammate access to the VM but doesn't allow the VM to access any other devices under the same subnet.

VLAN is a way to do it, but too complicated and require some hardwares and softwares.

The easiest way is to rely on the VM's firewall, simply don't give sudo access to the guest user.

sudo ufw enable
sudo ufw allow out to 192.168.0.1
sudo ufw deny out to 192.168.0.0/24

ufw allow out to 192.168.0.1 is for allowing traffic to router, otherwise it will not be able to connect to external network.

The ufw deny out to 192.168.0.0/24 must be run after ufw allow out to 192.168.0.1, as the rules are like a chain in iptables, if the deny rule comes first, traffic to router will be blocked and the allow rule won't even to reached.

The prepend keyword can be used to move a rule's priority ufw prepend deny out to 192.168.0.0/24.

The following script detects all submodules, cd into them and commit all changes.

Good for note taking submodules which don't require a very meaningful commit message.

Saves the trouble for commit every submodule separately.

Save the script as a shell script file update-all.sh and run it before every commit to the main repo.

#!/bin/bash
cwd=$PWD


submodulePaths=$(git submodule | awk '{$1=$1;print}' | cut -d ' ' -f 2)

function git_add_commit_update {
    nothing=$(git status | grep 'nothing to commit')
    if [[ -z $nothing ]]; then
        git add .
        git commit -m "Auto (Update Module)"
        git push
    fi
}

for path in $submodulePaths
do
    cd $path
    detached_head=$(git branch | grep 'HEAD detached')
    echo $path
    if [[ -z $detached_head ]]; then
        git_add_commit_update
    else
        git stash
        git checkout master
        git stash pop
        git_add_commit_update
    fi
    cd $cwd
done

Docs​

Tauri Updater

Bundler Artifacts has sample CI and config script.

Cross-Platform Compilation has a sample GitHub Action CI script for cross-platform compilation (Windows, MacOS and Linux). Compiled files are stored as artifacts in a draft GitHub release. The release assets will be read by updater server for auto-update.

Sample tauri.config.json

Building​

For updater to work, a public key is required.

"updater": {
    "active": true,
    "endpoints": [
        "https://releases.myapp.com/{{target}}/{{current_version}}"
    ],
    "dialog": true,
    "pubkey": "YOUR_UPDATER_SIGNATURE_PUBKEY_HERE"
}

A pair of keys can be generated with tauri signer generate -w ~/.tauri/ezup.key.

If update is configured, then private key and password environment variables must be set.

The following script can automatically load the private key as environment variable. Assuming password is an empty string.

#!/usr/bin/env bash
PRIVATE_KEY_PATH="$HOME/.tauri/ezup.key";
if test -f "$PRIVATE_KEY_PATH"; then
    export TAURI_PRIVATE_KEY=$(cat ~/.tauri/ezup.key); # if the private key is stored on disk
    export TAURI_KEY_PASSWORD="";
else
    echo "Warning: Private Key File Not Found";
fi

CICD (GitHub Action)​

In GitHub Action, environment variables can be set like this in the top level of yml file.

env:
  TAURI_PRIVATE_KEY: ${{ secrets.TAURI_PRIVATE_KEY }}
  TAURI_KEY_PASSWORD: ${{ secrets.TAURI_KEY_PASSWORD }}

I encountered a error during compilation on Ubuntu platform.

Error: thread '<unnamed>' panicked at 'Can't detect any appindicator library', src/build.rs:326:17

I found a solution in this issue.

Install libayatana-appindicator3-1-dev with apt for ubuntu.

Updater Server​

vercel/hazel is a updater server for electron, can be deployed in a few clicks on vercel.

lemarier/tauri-update-server forks vercel/hazel.

I forked lemarier/tauri-update-server to be HuakunShen/tauri-update-server.

The reason I made a fork is that, new upates were made in vercel/hazel, and I merged the new commits to lemarier/tauri-update-server.

With one click, an update server can be deployed on Vercel.

See EzUp and HuakunShen/tauri-ezup-updater for and example.

The former is the actual Tauri app. The later is the corresponding update server.