Build Your Own Top Programming Stories Dashboard

This challenge is to build a dashboard to consolidate the top programming, software engineering, AI or whatever your interests are from one or more online sources.

Much like the software teleprompter project, this is a project I’ve built to address my own needs. I want to consolidate multiple sources of stories into a single dashboard, filtering the stories so that it presents the set that are most likely to interest me.

This project is sponsored by Aiven who are providing free cloud Apache Kafka clusters for developers - it’s a great opportunity to learn about Apache Kafka and get experience building a project with it.

The Challenge - Building A Top Programming Stories Dashboard

For this challenge you’re going to build a tool to find stories on Hacker News ingest them into Kafka and then make the stories available via an API. On top of that API you’re going to build a web application to view the stories.

This project is a great want to learn about finding data sources, ingesting them into Kafka and then accessing them from Kafka using Kafka’s filtering. it’s a great project to practice building a small distributed system.

Step Zero

Set up your development environment ready to begin. Choose your programming language and grab a free Kafka cluster from Aiven. Install the Kafka client library for your chosen language. Set up your project structure with separate modules for the scraper, backend API, and a simple test setup. You might want to install curl, Postman or a similar tool for testing your API endpoints.

Step 1

In this step your goal is to build a simple Hacker News scraper that fetches stories from the API.

Here's how to fetch stories from Hacker News:

The API lives at https://hacker-news.firebaseio.com/v0/. All responses are JSON. Documentation for the API is available on their GitHub.

Different endpoints give you different story rankings:

• Top stories: https://hacker-news.firebaseio.com/v0/topstories.json
• New stories: https://hacker-news.firebaseio.com/v0/newstories.json
• Best stories: https://hacker-news.firebaseio.com/v0/beststories.json
• Ask HN: https://hacker-news.firebaseio.com/v0/askstories.json
• Show HN: https://hacker-news.firebaseio.com/v0/showstories.json

These return arrays of story IDs, like [12345, 12346, 12347, ...].

Once you have IDs, fetch each story: https://hacker-news.firebaseio.com/v0/item/12345.json

This returns details like:

json

{
  "by": "username",
  "id": 12345,
  "score": 142,
  "time": 1234567890,
  "title": "Something interesting",
  "type": "story",
  "url": "https://example.com"
}

Implement functions to poll the /topstories and /newstories endpoints, then fetch individual story details. Parse the JSON responses and create a story object/struct with fields like title, URL, score, author, and timestamp.

Ideally you should write some tests to verify your scraper correctly handles the Hacker News API format. For now, print the stories to console to verify everything works.

Step 2

In this step your goal is to set up your Kafka producer and publish stories to a topic. I suggest you try out Aiven’s free Kafka cluster so you can avoid installing, setting up and managing Kafka. Sign up for an account and create a new Kafka service (choose the free tier)

Once you have a Kafka cluster, follow Aiven’s instructions to connect to the cluster. create a topic called hn-stories, and configure your scraper to publish each story as a message to Kafka.

Decide on your message format (JSON is a good choice). Add a polling loop that fetches new stories every 60 seconds and publishes them to the hn-stories topic. Use Kafka's console consumer to verify messages are arriving correctly. Handle errors gracefully if the API or Kafka is unavailable.

Step 3

In this step your goal is to build a backend API service to consume from Kafka and exposes stories via HTTP. Create a simple REST API (Express, Flask, FastAPI, whatever fits your stack) with endpoints like GET /stories and GET /stories/top.

Your service should consume messages from the Kafka topic, store them in memory (a simple array or map is fine for now), and return them when the API is called. Add basic filtering by score threshold or time range. Test it with curl or Postman.

Step 4

In this step your goal is to add configurable consumer-side filtering so a story-api instance can be deployed to handle specific story types or keywords. It’s overkill for this solution, but if we were building a bigger distributed story system we might do this to create endpoints for specific story types.

Add configuration options to config.yaml to specify which story types, keywords, and minimum scores this instance should consume and store. Implement filtering logic in the consumer so stories matching the configured filters are stored in memory, while non-matching stories are discarded at ingest time. This enables running separate instances (e.g., story-api-ask for Ask HN stories, story-api-rust for Rust-related stories, story-api-top for high-score stories) that each store and serve only their relevant subset of stories. Each instance still exposes the full /stories API but returns only stories matching its configured filters, making the system more scalable and distributed.

Step 5

In this step your goal is to build a simple web frontend to display the stories.

Create a simple frontend that fetches stories from your backend API and displays them in a list. Implement auto-refresh so new stories appear without reloading the page. Keep it simple but functional.

Going Further

• Add Reddit support and multi-source aggregation.
• Extend your scraper to also fetch from Reddit's JSON API (pick a few subreddits like r/programming or r/technology).
• Publish Reddit posts to a separate Kafka topic or include a source field in your messages.
• Update your backend to consume from both topics and merge the results.
• Add source filtering to your API so users can choose to see only HN, only Reddit, or both.

Help Others by Sharing Your Solutions!

If you think your solution is an example other developers can learn from please share it, put it on GitHub, GitLab or elsewhere. Then let me know - ping me a message on the Discord Server, via Twitter or LinkedIn or just post about it there and tag me. Alternately please add a link to it in the Coding Challenges Shared Solutions Github repo.

Get The Challenges By Email

If you would like to receive the coding challenges by email, you can subscribe to the weekly newsletter on SubStack here: