MoreRSS

site iconThe Practical DeveloperModify

A constructive and inclusive social network for software developers.
Please copy the RSS to your reader, or quickly subscribe to:

Inoreader Feedly Follow Feedbin Local Reader

Rss preview of Blog of The Practical Developer

I built an idea-to-repo CLI with Github copilot

2026-02-14 20:36:44

I built idea-2-repo, a CLI that takes plain-English product ideas and turns them into a structured starter repository — complete with docs, TODOs, architecture outlines, and source scaffolding — by leveraging GitHub Copilot CLI’s natural-language coding agent directly in the terminal.

What I built

idea2repo aims to remove blank-project friction by generating a ready-to-use project skeleton from one simple idea prompt.

Core capabilities

✔ Idea normalization + project classification
✔ Architecture suggestion with Copilot CLI
✔ Scaffold generation (docs, TODOs, source files)
✔ Offline fallback (REASONING_BACKEND=offline)
✔ Dry-run mode for safe previews

1) Environment & Repo Context

I ensured tool versions and repository context were set up before generation.


2) Validation Checks

I verified the core validation pipeline end-to-end.

Lint & Build

Unit / Integration Tests

E2E Tests

3) Copilot CLI Usage

I confirmed Copilot CLI availability and used it to shape architecture ideas.

Copilot Version

Copilot Prompt Output

Prompt used:

copilot -p "Design a TypeScript CLI scaffold architecture in 5 concise bullets."

4) Real Project Generation

I ran a real project generation (non-dry run):

node dist/bin/idea2repo.js generate "AI-powered expense tracker for freelancers"

Generated artifacts verified:
✔ Project folder exists
✔ Docs and scaffold exist
✔ TODOs and architecture files present


5) Offline Fallback Demonstration

Verified offline mode behavior with:

REASONING_BACKEND=offline node dist/bin/idea2repo.js generate "Simple todo api for students" --dry-run

6) CLI UX / Help Output

CLI usability demonstration:

node dist/bin/idea2repo.js --help

What I Learned

Building idea2repo taught me a few practical lessons:

Copilot CLI shines at rapid architectural ideation when prompts are clear and scoped.

Offline fallback matters for reproducibility and deterministic runs.

Dry-run plus real-run improves confidence during demos and validation.

Repo

GitHub: https://github.com/GeoAziz/idea-2-repo

GitHubCopilot #CopilotCLI #DevChallenge #BuildInPublic

Fundamental matters more in AI era

2026-02-14 20:36:03

Large Language Models (LLMs) have become an inseparable part of the modern computing landscape. Software development is no exception. We’ve moved past the stage where AI only generates simple boilerplate; today, LLMs are capable of implementing complex logic and architecting entire applications.

Naturally, this raises a pressing question for those of us in the industry: Is there a future for software developers, or are we being phased out?

Shift is Real

Some might argue that "no-code" movements have always tried to replace developers, only to fail because a professional is always eventually needed. However, the current shift feels different. This is the most significant paradigm shift I’ve witnessed since I started earning a living through code.

Ignoring LLMs in favor of "pure" manual coding will soon lead to a dead end. But there is a flip side: if you simply rely on LLMs to write code while acting as a mere "code checker," you become easily replaceable.

Two Paths Forward: Strategy in the Age of Autopilot

To stay relevant, developers must choose distinct paths. The middle ground is rapidly disappearing.

The Architect of Experience : Use AI as a force multiplier to solve human problems. Your value lies in how quickly you can integrate APIs and LLMs to build trendy, user centric services.

The Architect of Systems (The Fundamentalist): Focus on the "under-the-hood" mechanics. As LLMs flood the world with abstraction, we need engineers who understand why a system fails under high concurrency or why a memory management strategy is suboptimal.

I recently came across a blog post that perfectly encapsulates these sentiments. To be honest, this article has been motivated from here.

https://notes.eatonphil.com/2026-01-19-llms-and-your-career.html

Don't Just Accept -> Validate

The greatest trap of the LLM era is complacency. We must not settle for what the LLM hands us. Think of it this way: the time you've saved by not having to manually search through documentation should be reinvested into verifying and understanding the generated logic. When you combine your hard-earned experience with the efficiency of an LLM, your competitiveness doesn't just increase—it multiplies. You aren't just a consumer of AI; you are its auditor and architect.

Opening the Black Box: Back to the Metal

The "Black Box" problem is a silent threat. In the data world, we stand on the shoulders of giants like Kubernetes, Spark, Flink, and Airflow. Yet, very few engineers understand these tools beyond their documentation.

We must remember a fundamental truth: No matter how sophisticated the AI or how complex the technology, everything eventually runs on CPU, Memory, and Network. When a critical issue occurs in production, the root cause is almost always found within these three pillars. Engineers who can bridge the gap between high level AI abstractions and these fundamental hardware constraints will never be out of demand.

The "Build Your Own X" Philosophy

This is exactly why I’ve been focusing on "build-your-own-x" projects. The goal is to bridge the gap between "using" a tool and "understanding" its core principles.

Interestingly, you don't need to go back to school to do this. Your LLM is the ultimate mentor. A modern LLM can drastically reduce the time it takes to grasp complex internal architectures. It shouldn't just write your code; it should explain the "why" behind the logic, acting as a teacher that uncovers what you didn't even know you were missing.

Stay Curious

While headlines talk about developer layoffs, there is still an immense demand for engineers who possess deep technical intuition and an insatiable curiosity.

The AI era doesn't mean we study less; it means we must study deeper. Even if LLMs handle the bulk of the coding, services built for humans will always require people who understand the soul of the machine. Don't let the convenience of AI stifle your technical curiosity—use that convenience to fuel it.

Reference

Self-Hosted Docker Registry on Your Own SFTP: Introducing Refity

2026-02-14 20:35:31

refity

If you want a private Docker registry but don't want to depend on a cloud registry or object storage, Refity might be what you need. It's a small, self-hosted registry that uses SFTP as the only backend for your images. Everything—blobs, manifests, metadata—lives on your SFTP server. No S3, no GCS, no vendor lock-in.

I built Refity for myself that already have SFTP (rent from Hetzner Storage Box) and want to reuse it for container images without adding another storage system.

Why SFTP?

  • You already have SFTP (e.g. Hetzner Storage Box, shared hosting, or your own server).
  • You want to avoid cloud object storage and its pricing/complexity.
  • You need something that works in locked-down or legacy environments where only SSH/SFTP is allowed.
  • You prefer one place for backups and access control (your existing SFTP).

Refity doesn't replace your SFTP; it uses it as the only storage backend.

What Refity Does

  • Docker Registry API v2 — Standard docker push and docker pull; works with existing tools and CI.
  • SFTP-only storage — All layers and manifests are stored on your SFTP server; the app only coordinates and serves them.
  • Web UI — A React app to browse groups, repositories, and tags, create/delete repos, and copy docker pull commands.
  • Auth — JWT-based login; you can put the backend behind your own proxy/VPN for extra security.
  • One-command deploy — Backend + frontend with Docker Compose; optional env vars for production (JWT secret, CORS, SSH host verification).

So: a classic "registry in front of your own storage" with a simple UI and no new storage stack.

Quick Start

1. Clone and configure

git clone https://github.com/troke12/refity.git
cd refity
cp .env.example .env

Edit .env with your SFTP details and (for production) a JWT_SECRET:

FTP_HOST=your-sftp.example.com
FTP_PORT=22
FTP_USERNAME=sftpuser
FTP_PASSWORD=sftppass
JWT_SECRET=your-long-random-secret   # required in production

2. Run with Docker Compose

docker-compose up -d

3. Use it

Push an image:

docker tag nginx localhost:5000/mygroup/nginx:latest
docker push localhost:5000/mygroup/nginx:latest

Pull:

docker pull localhost:5000/mygroup/nginx:latest

Groups and repositories are created from the UI (or you can align with how you organize folders on SFTP). No auto-creation of top-level paths; you keep control over structure and access.

Tech Stack

  • Backend: Go — Registry v2 + REST API, SQLite for metadata, JWT auth.
  • Frontend: React + Vite, served by nginx in Docker.
  • Images: troke12/refity-backend and troke12/refity-frontend on Docker Hub (tags: latest, v1.0.0).

Who It's For

  • Small teams or solo devs who want a private registry without cloud dependencies.
  • Environments where SFTP is the only or preferred storage (e.g. existing storage boxes or restricted networks).
  • Anyone who likes to self-host and keep image storage under their control.

Links

If you try it and have feedback or ideas, open an issue or discussion on GitHub.

Converting Large JSON, NDJSON, CSV and XML Files without Blowing Up Memory

2026-02-14 20:25:13

Most of us have written something like this at some point:

const data = JSON.parse(hugeString);

It works.

Until it doesn't.

At some point the file grows.\
50MB. 200MB. 1GB. 5GB.

And suddenly:

  • The tab freezes (in the browser)
  • Memory spikes
  • The process crashes
  • Or worse --- everything technically "works" but becomes unusable

This isn't a JavaScript problem.

It's a buffering problem.

The Real Issue: Buffering vs Streaming

Most parsing libraries operate in buffer mode:

  • Read the entire file into memory
  • Parse it completely
  • Return the result

That means memory usage scales with file size.

Streaming flips the model:

  • Read chunks
  • Process incrementally
  • Emit records progressively
  • Keep memory nearly constant

That architectural difference matters far more than micro-optimizations.

Why I Built a Streaming Converter

I've been working on a project called convert-buddy-js, a Rust-based
streaming conversion engine compiled to WebAssembly and exposed as a
JavaScript library.

It supports:

  • XML
  • CSV
  • JSON
  • NDJSON

The core goal was simple:

Keep memory usage flat, even as file size grows.

Not "be the fastest library ever."\
Just predictable. Stable. Bounded.

What Does "Low Memory" Actually Mean?

Here's an example from benchmarks converting XML → JSON.

Scenario Tool File Size Memory Usage
xml-large convert-buddy 38.41 MB ~0 MB change
xml-large fast-xml-parser 38.41 MB 377 MB

The difference is architectural.

The streaming engine processes elements incrementally instead of
constructing large intermediate structures.

CSV → JSON Benchmarks

I benchmarked against:

  • PapaParse\
  • csv-parse\
  • fast-csv

Here's a representative neutral case (1.26 MB CSV):

Tool Throughput
convert-buddy 75.96 MB/s
csv-parse 22.13 MB/s
PapaParse 19.57 MB/s
fast-csv 15.65 MB/s

In favorable large cases (13.52 MB CSV):

Tool Throughput
convert-buddy 91.88 MB/s
csv-parse 30.68 MB/s
PapaParse 24.69 MB/s
fast-csv 19.68 MB/s

In most CSV scenarios tested, the streaming approach resulted in roughly
3x--4x throughput improvements, with dramatically lower memory overhead.

Where Streaming Isn't Always Faster

For tiny NDJSON files, native JSON parsing can be faster.

Scenario Tool Throughput
NDJSON tiny Native JSON 27.10 MB/s
NDJSON tiny convert-buddy 10.81 MB/s

That's expected.

When files are extremely small, the overhead of streaming infrastructure
can outweigh benefits.\
Native JSON.parse is heavily optimized in engines and extremely
efficient for small payloads.

The goal here isn't to replace native JSON for everything.

It's to handle realistic and large workloads predictably.

NDJSON → JSON Performance

For medium nested NDJSON datasets:

Tool Throughput
convert-buddy 221.79 MB/s
Native JSON 136.84 MB/s

That's where streaming and incremental transformation shine ---
especially when the workload involves structured transformation rather
than just parsing.

What the Library Looks Like

Install:

npm install convert-buddy-js

Then:

import { convert } from "convert-buddy-js";

const csv = 'name,age,city\nAlice,30,NYC\nBob,25,LA\nCarol,35,SF';

// Configure only what you need. Here we output NDJSON.
const buddy = new ConvertBuddy({ outputFormat: 'ndjson' });

// Stream conversion: records are emitted in batches.
const controller = buddy.stream(csv, {
  recordBatchSize: 2,

  // onRecords can be async: await inside it if you need (I/O, UI updates, writes...)
  onRecords: async (ctrl, records, stats, total) => {
    console.log('Batch received:', records);

    // Simulate slow async work (writing, rendering, uploading, etc.)
    await new Promise(r => setTimeout(r, 50));

    // Report progress (ctrl.* is the most reliable live state)
    console.log(
      `Progress: ${ctrl.recordCount} records, ${stats.throughputMbPerSec.toFixed(2)} MB/s`
    );
  },

  onDone: (final) => console.log('Done:', final),

  // Enable profiling stats (throughput, latency, memory estimates, etc.)
  profile: true
});

// Optional: await final stats / completion
const final = await controller.done;
console.log('Final stats:', final);

It works in:

  • Node
  • Browser
  • Web Workers

Because the core engine is written in Rust and compiled to WebAssembly.

Why Rust + WebAssembly?

Not because it's trendy.

Because:

  • Predictable memory behavior
  • Strong streaming primitives
  • Deterministic performance
  • Easier control over allocations

WebAssembly allows that engine to run safely in the browser without
server uploads.

When This Tool Makes Sense

You probably don't need it if:

  • Files are always < 1MB
  • You're already happy with JSON.parse
  • You don't care about memory spikes

It makes sense if:

  • You process large CSV exports
  • You handle XML feeds
  • You work with NDJSON streams
  • You need conversion in the browser without uploads
  • You want predictable memory footprint

What I Learned Building It

  • Streaming is not just about speed --- it's about stability.
  • Benchmarks should include losses.
  • Native JSON.parse is hard to beat for tiny payloads.
  • Memory predictability matters more than peak throughput.

Closing Thoughts

There are many good parsing libraries in the JavaScript ecosystem.

PapaParse is mature.\
csv-parse is robust.\
Native JSON.parse is extremely optimized.

convert-buddy-js is simply an option focused on:

  • Streaming
  • Low memory usage
  • Format transformation
  • Large file handling

If that matches your constraints, it may be useful.

If not, the ecosystem already has excellent tools.

If you're curious, the full benchmarks and scenarios are available in
the repository.
convert-buddy-js — npm
brunohanss/convert-buddy

And if you have workloads where streaming would make a difference, I’d be interested in feedback.
You can get more information or try the interactive browser playground here: https://convert-buddy.app/

How We Run Lighthouse from 18 Regions in Under 2 Minutes

2026-02-14 20:19:08

Most performance monitoring tools test your site from one location, or run tests sequentially across regions. That means testing from 18 locations can take 20+ minutes.

We needed something faster. Ahoj Metrics tests from 18 global regions simultaneously in about 2 minutes. Here's how.

The Architecture

The core idea is simple: don't keep workers running. Spawn them on demand, run the test, destroy them.

We use Fly.io's Machines API to create ephemeral containers in specific regions. Each container runs a single Lighthouse audit, sends the results back via webhook, and destroys itself.

Here's how a request flows through the system:

The key design decision: one audit = one ReportRequest, regardless of how many regions you test. Test from 1 region or 18 - it's the same user action.

Spawning Machines with the Fly.io API

Here's the actual code that creates a machine in a specific region:

class FlyMachinesService
  API_BASE_URL = "https://api.machines.dev/v1"

  def self.create_machine(region:, env:, app_name:)
    url = "#{API_BASE_URL}/apps/#{app_name}/machines"

    body = {
      region: region,
      config: {
        image: ENV.fetch("WORKER_IMAGE", "registry.fly.io/am-worker:latest"),
        size: "performance-8x",
        auto_destroy: true,
        restart: { policy: "no" },
        stop_config: {
          timeout: "30s",
          signal: "SIGTERM"
        },
        env: env,
        services: []
      }
    }

    response = HTTParty.post(
      url,
      headers: headers,
      body: body.to_json,
      timeout: 30
    )

    if response.success?
      Response.new(success: true, data: response.parsed_response)
    else
      Response.new(
        success: false,
        error: "API error: #{response.code} - #{response.body}"
      )
    end
  end
end

A few things worth noting:

auto_destroy: true is the magic. The machine cleans itself up after the process exits. No lingering containers, no zombie workers, no cleanup cron jobs.

performance-8x gives us 4 vCPU and 8GB RAM. Lighthouse is resource-hungry - it runs a full Chrome instance. Underpowered machines produce inconsistent scores because Chrome competes for CPU time. We tried smaller sizes and the variance was too high.

restart: { policy: "no" } means if Lighthouse crashes, the machine just dies. We handle the failure on the Rails side by checking for timed-out reports.

services: [] means no public ports. The worker doesn't need to accept incoming traffic. It runs Lighthouse and POSTs results back to our API. That's it.

The Worker

Each Fly.io machine runs a Docker container that does roughly this:

  1. Read environment variables (target URL, callback URL, report ID)
  2. Launch headless Chrome
  3. Run Lighthouse audit
  4. POST the JSON results back to the Rails API
  5. Exit (machine auto-destroys)

The callback is a simple webhook. The worker doesn't need to know anything about our database, user accounts, or billing. It just runs a test and reports back.

Handling Results

On the Rails side, each Report record tracks its own status:

class ReportRequest < ApplicationRecord
  has_many :reports

  def check_completion!
    return unless reports.all?(&:completed?)

    update!(status: "completed")
    update_cached_stats!
    check_monitor_alert if site_monitor.present?
  end
end

When a worker POSTs results, the corresponding Report is updated. After each update, we check if all reports for the request are done. If so, we aggregate the results, calculate averages, and update the dashboard.

Each report is independent. If the Sydney worker fails but the other 17 succeed, you still get 17 results. The failed region shows as an error without blocking everything else.

Cost Math

This is the part that makes ephemeral workers compelling. Compare two approaches:

Persistent workers (18 regions, always-on):

  • 18 machines running 24/7
  • Even the smallest Fly.io machine is ~$5/month
  • That's $90/month minimum, mostly sitting idle

Ephemeral workers (our approach):

  • Machines run for ~2 minutes per audit
  • performance-8x costs roughly $0.0001344/second
  • One 18-region audit costs about $0.29
  • 100 audits/month = ~$29

At low volume, ephemeral is dramatically cheaper. The crossover point where persistent workers become more cost-effective is well beyond our current scale.

The tradeoff is cold start time. Each machine takes a few seconds to boot. For our use case (users expect a 1-2 minute wait anyway), that's invisible.

The Background Job Layer

We use Solid Queue (Rails 8's built-in job backend) for everything. No Redis, no Sidekiq.

# config/recurring.yml
production:
  monitor_scheduler:
    class: MonitorSchedulerJob
    queue: default
    schedule: every minute

The MonitorSchedulerJob runs every minute, checks which monitors are due for testing, and kicks off the Fly.io machine spawning. Monitor runs are background operations - they don't count toward the user's audit quota.

This keeps the architecture simple. One PostgreSQL database handles the queue (via Solid Queue), the application data, and the cache. No Redis to manage, no separate queue infrastructure to monitor.

What We Learned

Lighthouse needs consistent resources. When we first used shared-cpu machines, scores would vary by 15-20 points between runs of the same URL. Bumping to performance-8x brought variance down to 2-3 points. The extra cost per audit is worth the consistency.

Timeouts need multiple layers. We set timeouts at the HTTP level (30s for API calls), the machine level (stop_config timeout), and the application level (mark reports as failed after 5 minutes). Belt and suspenders.

Region availability isn't guaranteed. Sometimes a Fly.io region is temporarily unavailable. We handle this gracefully - the report for that region shows an error, but the rest of the audit completes normally.

Webhook delivery can fail. If our API is temporarily unreachable when the worker finishes, we lose the result. We're adding a retry mechanism and considering having workers write results to object storage as a fallback.

The Numbers

After running this in production since January 2026:

  • Average audit time: ~2 minutes (single region or all 18)
  • P95 audit time: ~3 minutes
  • Machine boot time: 3-8 seconds depending on region
  • Success rate: ~97% (3% are timeouts or region availability issues)
  • Cost per audit: $0.01-0.29 depending on regions selected

Try It

You can test this yourself at ahojmetrics.com. Free tier gives you 20 audits/month - enough to see how your site performs from Sydney, Tokyo, Sao Paulo, London, and more.

If you have questions about the architecture, ask in the comments. Happy to go deeper on any part of this.

Built with Rails 8.1, Solid Queue, Fly.io Machines API, and PostgreSQL. Frontend is React + TypeScript on Cloudflare Pages.

Slides Narration Video Best Practices (2Slides + Remotion) — From One Prompt to a Ready-to-Share MP4

2026-02-14 20:18:35

Slides narration video best practices cover

Slides Narration Video Best Practices (2Slides + Remotion) — From One Prompt to a Ready-to-Share MP4

Creating a slides narration video used to be a multi-hour workflow: outline → design slides → write a script → record voiceover → sync timing → render.

With 2Slides + Remotion, you can now do it in one prompt and get a clean MP4 in ~10 minutes—good enough for product explainers, training, and marketing content.

This guide is a practical checklist for shipping higher-quality narration videos consistently.

What you’re building

Slides + voice + Remotion pipeline diagram

A fully automated pipeline:

1) Generate slides (pages/PDF) with 2Slides
2) Generate narration audio with 2Slides
3) Download assets (pages + audio zip)
4) Create a Remotion project and sequence pages + audio
5) Render MP4

The 80/20: what makes narration videos feel “professional”

1) Script first, slides second (even if AI generates both)

A narration video is audio-led. If the narration is clear and well-paced, viewers forgive simple slides. If narration is messy, no amount of design saves it.

Best practices

  • Keep sentences short (spoken language, not blog language)
  • One idea per slide
  • Avoid reading dense paragraphs verbatim

2) Timing rules that never fail

  • Target 12–18 seconds per slide for explainers
  • Narration pace: ~140–170 wpm (English)
  • If narration for a slide exceeds 25–30s, split the slide

3) Visual rhythm: treat slides like cuts

A narration video should feel like “cuts” in video editing.

  • Alternate layouts (text-heavy → visual → text)
  • Insert an occasional “breathing slide” (a big headline + 1 icon)
  • Use consistent margins and font hierarchy

4) Audio quality: loudness and silence

Even with great TTS, you still want predictable audio levels.

  • Normalize to a consistent loudness target (e.g. -16 LUFS for web)
  • Add 200–500ms padding between slides when switching topics
  • Avoid abrupt starts: use a short fade-in (50–100ms)

5) Accessibility: subtitles are a multiplier

If the workflow can produce captions, do it.

  • Burn-in captions for social distribution
  • Or ship a sidecar .srt file

Recommended workflow in Claude Code + OpenClaw (copy/paste)

You can run this workflow from an agent environment (Claude Code / OpenClaw) by installing two skill packs:

Then use a single prompt like this (replace the topic):

Please help me create a slides narration video for the topic: [YOUR_TOPIC_CONTENT]

- Create slides using 2Slides create pdf slides API; decide the slide pages count by AI.
- Generate voice narration using 2Slides API.
- Download slides pages and voices zip using 2Slides API.
- Create a Remotion project and a slides narration video using pages and voices in sequence.
- Render and output the video.

Real case demo:
https://www.youtube.com/watch?v=_KswiI-Tgdc

Best practices for each pipeline step

Step 1) Slide generation (pages/PDF)

Goal: create slides that are readable at video resolution.

  • Prefer larger fonts, fewer bullets
  • Avoid tiny charts (video compression kills them)
  • Use 16:9 layouts, high contrast

Pro tip: ask the agent to choose slide count based on content length.

Step 2) Voice narration

Goal: match narration structure to slide boundaries.

  • Generate narration per slide (not one giant audio)
  • Enforce a consistent tone ("confident, calm, concise")
  • For acronyms/brand terms: provide pronunciation hints

Step 3) Asset packaging (zip download)

Goal: make rendering deterministic.

  • Name assets in order: 001.jpg, 002.jpg… and 001.mp3, 002.mp3
  • Keep everything in a single folder for Remotion

Step 4) Remotion sequencing

Goal: correct sync, zero glitches.

  • Set slide duration from audio length + padding
  • Crossfade (optional): 6–12 frames is enough
  • Consider a subtle background music bed at -28 to -32 dB

Step 5) Rendering

Goal: export a share-ready MP4.

  • Use H.264, 1080p, ~8–12 Mbps for most use cases
  • Validate on mobile (common failure: text too small)

QA checklist (before you ship)

Slides narration video QA checklist

  • Does each slide’s headline match the narration?
  • Any slide stays on screen for >25–30s? Split it.
  • Any slide shows tiny text? Increase font size.
  • Any abrupt audio start? Add a short fade/padding.
  • Add subtitles for social distribution.

Next step

Try it with a real topic you care about (a product feature, onboarding flow, or weekly update). If you can turn one prompt into a consistent video pipeline, you can ship content faster than your competitors.