Hello Claw – Setting up a local agentic assistant to level up my learning, projects, and logistics

I’ve been experimenting with AI tools for awhile – Claude Code (and more claude code), ChatGPT, Copilot, Cursor, Windsurf. Especially with the fourth baby here, I have a lot of time where I’m up-at-all hours with a hand free to use my phone while I feed a snoozy baby. I’m a huge home automation fan (I didn’t blog about it – but my 2024 project was a modified voice assistant running on home assistant OS with custom hardware backed by a local LLM; on top of that I have a parallel Alexa-based automation stack). And during my postpartum period I’m brushing up on my AI fundamentals and the latest with genAI.

That’s all to say, that the  OpenClaw project – which promises an easy phone-chat interface to all things AI x making and a great integration with automation, was the ultimate nerd snipe for this season of my life. Dutifully, last week I bought a Mac Mini, and paid the extra $9 for it to arrive at my doorstep that day. Do you need a mac mini for this? You do not. Do you want a mac mini for this? If you have more money to burn than time, then maybe.

Sidebar: Why I went with the Mini instead of some lobster cloud

Some of these justifications are tenuous. What it boils down to probably, is

  • I love apple gadgets (give me any excuse to give Apple my credit card)
  • Reducing friction to do projects is compelling to me whatever the cost. (I have two office spaces, and this would mean I could have an always attached computer in my second)

But technically I did have a few thoughts

  • For my modified-voice-assistant project in 2024 I ran some local LLMs and the prospect of eventually having dedicated easy to use hardware for that was juicy.
  • On top of that, the idea that I’d have access to the bare metal and have complete control on it in my environment security wise was appealing.
  • The IP on the browser automation coming from a home network vs a server farm is apparently advantageous for not getting the agents blocked.

So I sprung for the mini

The Setup

I plugged in the mini and ran a script to install. My basic principle was to ensure that OpenClaw always had its own dedicated accounts, and not to share any of my personal account information with the system, as well as to keep everything to my local network only. What I needed was new dedicated accounts for

  • Claw Google Suite
  • Claw Phone number (got a $5/month, registered to my phone)
  • Claw WhatsApp (was able to setup a separate App: whatsApp business and register to the Claw Phone)
  • Claw Password manager
  • Claw Apple ID
  • Claw Anthropic Key
  • Claw Amazon Business Account
  • Claw Dropbox account

I setup with WhatsApp (straightforward) and iMessage (a bit more involved). I made sure that they would just accept messages from myself, my husband, and our nanny. I gave the AI a name. I prefer WhatsApp because its setup allows you to see when the Claw is thinking (via a … typing indicator which apparently is inaccessible in the iMessage config).

The Tools out of the Box

I didn’t enable many standard tools, preferring to write my own that I’d have some control over

The Tools I added

Adding a new capability generally involved chatting with the agent via whatsapp and asking it to do something new! You can also interface via a browser if you are at a computer. I added a number of things this way. The Claw runs via a browser separate from my regular chrome that it can interact with. So if I log in there, or it has credentials to log in itself, then it is good to automate anything a human would click through.

  • Making reservations at our local botanical gardens is a website nightmare. I have a skill now that you request a reservation and it makes it through browser automation
  • Hooked into home assistant
  • Voice message transcription
  • Installed CLI to interact with Google Drive (from Claw account; I shared my own calendar with it as read only)

Amazon Purchasing

This was a big one and took me awhile to setup correctly. The big hack was the Amazon business account. I needed a corporation lying around for this; but we share one for the household already, As needed you could set one up via Stripe Atlas but probably not worth the hassle unless you have other needs for it as we do. I wanted our assistant to be able to search for products, place orders, manage Subscribe & Save. The kind of stuff that’s not hard but eats time when you’re doing it three times a week with four kids burning through supplies.

Once I had the Amazon Business account, I was able to link my household Prime to it. I added the Claw, my nanny, and my husband as purchasers with myself as approver. I had to use emails not already associated with an Amazon account.This would mean that any of them could purchase themselves (and it would get routed for approval) with these new accounts, or they could ask the Claw via chat any questions about Claw’s orders or to purchase agentically on their behalf. I migrated our family’s subscribe and saves to the Claw’s account.

To test the whole pipeline, I asked the claw to list my subscribe and saves. It did. I then asked it to order a lobster plushie! It sent me a few screenshots and I selected one. It sent me an email, I clicked approve and I was good to go. With the approval workflow tested, I’m now confident asking it to buy a bunch of things agentically on my behalf without checking in as I know I’d get the chance to do a final review and click for accuracy and budget.

Blogging

I had it draft this blog by prompting “hey can you draft up what we’ve done as a blog and put it in a google doc and share with me”. I then didn’t directly use the text, because it sounds like an LLM and is pretty dry, but it did provide me nice reminders about what I might want to chat about.

A week in

I’m expanding my reach and starting with some fun stuff. I’ve had the Claw updated its “soul.md” file – aka its personality, and had it start sending a monthly self portrait (the featured image above). I’m now branching out into using it for curating a feed for me by combing what’s interesting on arXiv. Its pick for today was especially meta

I’m also working on plugging in my Home Assistant to my Claw via voice. Doing so securely will be a slight adventure. I’m also expanding its memory architecture and giving it access to more development tooling to agentically code and deploy. I’m excited to share what I make and how the assistant integrates into daily life as a busy parent, professional and maker.