Feed aggregator

Ask HN: Where do you go to learn data engineering patterns?

Hacker News - Thu, 11/14/2024 - 12:26pm

I am working on refactoring a system which I feel does some frequently encountered flows across organisations. Collects raw event streams, grabs unique URLs, crawls, extracts, creates vectors, uses these output in some downstream analysis and objects.

I have many different solutions which “work” but I am not sure which is optimal.

I find both the internet and LLMs to be a very poor resource for this type of work.

So where does one learn about how others have implemented such systems? Happy to reads books, blogs, repos!

Comments URL: https://news.ycombinator.com/item?id=42138619

Points: 1

# Comments: 0

Categories: Hacker News

The Poor UX/UI of Major Cloud Providers: ECS/VMs Management

Hacker News - Thu, 11/14/2024 - 12:26pm

The user experience of major cloud providers is surprisingly poor, especially when it comes to basic VM/instance management. Here are some major pain points I've encountered:

1. Complex Regional Segregation (AWS) - Instances are strictly segregated by regions - No unified view to manage instances across all regions by default (Need specific queries just to get a global view) - Sometimes you lose track of which region your instances are in

2. Instance Metadata Management is Unnecessarily Difficult - Can't easily rename instances (Google Cloud requires shutdown and restart) - No automatic tracking of which IAM account created the instance - Limited ability to add notes or tags for running services - When an instance runs multiple services (e.g., "service-A" actually running A, B, and C), there's no easy way to update its name or add proper documentation - Have to maintain external documentation to track what's running where - Need to SSH and check screen/tmux sessions to see what's actually running

3. SSH Key Management Issues (especially Google Cloud) - Can't properly manage SSH public keys - No ability to add descriptions to keys - Can only see a list of root public keys without context - No tracking of who added which key - No "last used" timestamp (unlike GitHub which handles this well)

I do use serverless solutions like Lambda/Cloud Functions and Firebase, but there are always cases where you need ECS/VMs. These basic UX issues make daily operations unnecessarily complicated.

What's your experience with cloud providers' UI/UX? Have you found any good solutions to these problems?

Comments URL: https://news.ycombinator.com/item?id=42138608

Points: 1

# Comments: 0

Categories: Hacker News

BoxLang

Hacker News - Thu, 11/14/2024 - 12:23pm

Article URL: https://www.boxlang.io/

Comments URL: https://news.ycombinator.com/item?id=42138572

Points: 1

# Comments: 0

Categories: Hacker News

These Bose Ultra Earbuds Are Back to Their Lowest Price Ever Ahead of Black Friday

CNET Feed - Thu, 11/14/2024 - 12:16pm
The only thing our reviewer didn't like about these uniquely designed Bose earbuds was the price, and right now they're $50 off.
Categories: CNET

Pages