Backend engineering

Language-agnostic concepts a Backend Engineer should know?

77 points by nicksont 12 hours ago | hide | past | favorite | 51 comments Hi, what concepts do you think a backend engineer should know which are language agnostic/independent? An example could be knowing the different ways of how to connect to the database from the backend such as ORM or basic SQL and their pros and cons?

danpalmer 10 hours ago [–]

I think it’s important to know about different data storage options and their trade-offs. Managing state is one of the hardest parts of backend development, particularly at scale, so an understanding of the trade-offs in databases/caches/blob-storage/queues, and when each is useful is important. I’d pay close attention to speed and “correctness”. What’s the consistency model of a system? Can we lose data and if so how? What’s the throughput? Latency?

These help choose good technology for backend systems, and helps answer questions like:

• Can we do this in-band while serving a user request?

• Can we do it 100 times to serve a request?

• If it completes successfully can we trust it or do we still need to handle failure?

• Can we trust it immediately or eventually?

There are lots of technologies and terms for all of this but I’ve specifically avoided them because the important bit is the mental model of how these things fit together and the things they allow/prevent.

reply

SOLAR_FIELDS 10 hours ago [–]

Absolutely this. Also, I’ve had to explain this to so many other engineers, both junior and senior to me: most data is inherently relational. This next statement is a bit opinionated: 9 times out of 10 you probably want an RDBMS. I’ve seen so many attempts to shoehorn some ElasticSearch/Mongo/Neo4j/whatever database into a design because the developer wanted to work on CoolDatabaseTech. Then you’re stuck dealing with joins in CoolDatabase that it wasn’t really designed to do and frustrated at CoolDatabase’s lack of drivers in X language. Later on you’re dealing with stability and scalability issues you would never see with BattleTestedRDBMS.

The amount of capability a well designed Postgres instance can output is insane. I’ve seen a single vertically scaled Postgres instance compete with 100+ node Spark clusters on computations.

reply

danpalmer 9 hours ago [–]

Exactly, but it goes a lot further than RDBMSs. For example does the application expect that all items on a queue will be processed? If so then you need a durable queue and Redis probably isn’t a good idea, and this will likely reduce the throughput of the queue which might change how it needs to be used. One I’ve been bitten by several times is expecting APIs to allow me to read-my-writes, only to find that their underlying data store is eventually consistent. The integration point/API client on our end may end up being twice as complex or more just to handle that.

reply

mamcx 8 hours ago [–]

9 times out of 10 you probably want an RDBMS. And the last 1 can be done (modeled) in a RDBMS when the scale/pressure/volume is low. In other words, wait until you feel the heat.

reply

SOLAR_FIELDS 8 hours ago [–]

Yeah, Postgres off the top of my head does NoSQL (JSONB), graph and time series stuff either natively or through some cheap or freely available add ons. It really can do anything. It’s not gonna be the best at that non-relational stuff, but it will do a “good enough” job for most use cases until you introduce heavy scaling. reply

Zealotux 10 hours ago [–]

I’m currently learning back-end coming from a front-end career, and I started reading “Designing Data-Intensive Applications” by Martin Kleppmann, seems to be a must-have for anyone who wants to get serious in this field. reply

davidkell 10 hours ago [–]

+1 For this recommendation, an exceptional book. In particular chapters 2-3 demystify how databases work, and provides an intuition on when to use different kinds of databases and query languages without the typical hype (spoiler alert - use an RDBMS!) reply

humbleMouse 6 hours ago [–]

Yes this is a great book and a perfect example of language agnostic knowledge that is useful. reply

throwaway81523 10 hours ago [–]

Security mindset: read the book Security Engineering (it is online), less for specific technical info than for the many war stories etc. which will help you guard against vulnerabilities and unforeseen consequences. Basics of cryptography: there are many dumb errors to avoid.

Antirez’s general advice about “10x programmers” is good: http://antirez.com/news/112

Thorough (not just basic) knowledge of SQL, if you don’t count that as a language. The sqlite.org “technical and design documents” about sqlite’s virtual machine and its query planner are well worth reading, and apply to other databases as well. ORM’s are less important than SQL, and are usually language specific as someone mentioned.

Reasonable clue about socket programming, even if you’re doing everything with libraries that wrap the details.

Comfort using debugging and profiling tools.

Lots of other stuff, I’m sure.

reply

comprev 10 hours ago [–]

This book? https://www.cl.cam.ac.uk/~rja14/book.html reply

throwaway81523 9 hours ago [–]

Yes. It is a little bit old by now, but it is a great book. reply

chris_j 2 hours ago [–]

Understand the importance of having good visibility of your system. Implement good logging and collect metrics, for example the four golden signals of throughput, latency, saturation and error rate. The Google SRE book gives a good introduction to some of these concepts. See for example https://sre.google/sre-book/monitoring-distributed-systems/. Understand how to load test your system and to reason about its behaviour under load and its failure modes when you push it too hard. It’s one thing to be able to build a system and functionally test it such that you’re confident that it behaves correctly when you send one request at a time. It’s another thing to let thousands or millions of real users hit it for real in production and to have confidence that you are giving them all a good experience.

reply

tekstar 11 hours ago [–]

C and how to debug it. If you understand the system a layer of abstraction or two below the layer you work in, you will be able to debug deeper. Learn system calls, Various ways how to examine processes.

I learnt a lot of this back in the day by completing war games on a site called digital evolution (dievo). Those are antiquated now but still a really fun way to learn it.

reply

pdpi 10 hours ago [–]

Also, learn Javascript and browser tech in general. One of the keys to high-performance systems is figuring out how to make the layer above and the layer below talk to each other while removing yourself from the process as much as possible. reply

cudgy 9 hours ago [–]

I would think that most front end developers already know JavaScript. It might make sense to look at some other approaches like Go (and maybe Rust/C/C++), which is designed for back-end work unlike JavaScript which was basically shoehorned into back-end work in order to leverage all the JavaScript developers in many companies. reply

tifadg1 10 hours ago [–]

sounds very niche from an employability PoV. reply

sidlls 10 hours ago [–]

These skills have helped me at every single job I’ve had, including my current one, where I’ve employed the skill a couple of times to discover bugs or unintended side-effects in third-party library code. For context I’ve worked in the industry for about 15 years now, in roles ranging from low-level programming on real-time systems to (currently) high level machine learning work. It might be niche, but it can also be a differentiator—even if I’m not the fastest coder or the best architect I have these other skills that make me valuable at critical times. That’s worth something anywhere.

And it’s not even knowledge of C or C++, or syscalls or whatever: it’s just basic “use a debugger” (not merely pantomime the commands, but understand what’s going on) skills that can be the real game changer.

reply

SOLAR_FIELDS 10 hours ago [–]

Totally agree with this. I have never written C or C++ in a professional manner but just knowing how to debug and compile it has proved to be an invaluable skill. Lots of software ecosystems often have some underlying C/C++ code that they are calling out to and being able to dive into that when there is an issue is an incredible skill to have. Make, Ninja etc. are fairly straightforward compared to something like Gradle and just knowing my way around that and Clang/GnuCC has gotten me a lot farther in my career.

reply

pdpi 10 hours ago [–]

It doesn’t make you “employable”. It does make you incredibly valuable once you’re inside. Being able to debug systems and being able to talk to other teams working on different levels of abstraction are both incredibly important skills. reply

tekstar 10 hours ago [–]

Yeah it’s the complete opposite of niche. It’s the fundamental understanding of processes, signals, syscalls, memory that every modern computing system is based upon. reply

fennecfoxen 10 hours ago [–]

You are by all means welcome to remain ignorant of how computers and frameworks and libraries all work, incapable of tracking down and fixing bugs in your application stack that affect you. Build a career filling in the boilerplate that Spring or Rails or whatever generates, and when the system doesn’t perform as you expected it to, throw up your hands and say “I don’t know what it’s doing,” and end the matter there. This is, presumably, a quite ordinary and common approach to software development. But if you do so, then I must ask: what are you doing on this site? What can you possibly get out of it?

reply

cudgy 9 hours ago [–]

Your first paragraph is dead on. When I first encountered Ruby on rails after developing in C++ for years, it was very difficult to accept the black box of ruby on rails. In fact, my instincts are to fear tools like that. The sheer number of dependencies on third parties when using tools like Ruby on rails is downright frightening. Good luck making sure that all of the libraries your linking to are secure and that you can fix any issues that come up. reply

konart 10 hours ago [–]

frameworks and libraries all work, incapable of tracking down and fixing bugs in your application stack… What does any of this has to do with C?

What can you possibly get out of it?

I hope you realise that many of commenters here are not even software engineers. HN covers more topics than just ‘another arrogant software dev lectures someone on topic X’

reply

whynaut 8 hours ago [–]

‘another arrogant software dev lectures someone on topic X’ Not GP. But this is explicitly a software thread.

frameworks and libraries all work, incapable of tracking down and fixing bugs in your application stack…

Pretty sure they are implying that C/C++ lies somewhere in most software stacks.

reply

konart 4 hours ago [–]

Pretty sure they are implying that C/C++ lies somewhere in most software stacks. So do many other things. It’s nice to know every bit of tech behind the scenes but one has only this much time.

reply

jgwil2 10 hours ago [–]

Well, one doesn’t have time to learn everything, so it’s entirely reasonable to do a cost-benefit analysis (in terms of employability or personal interest) before investing. reply

pdpi 9 hours ago [–]

The OP’s statement makes the context clear though: It’s not about knowing everything. It’s about knowing a couple of abstraction layers under you (and, as I argued elsewhere, above you too). It’s useful to understand a few classic networking problems that a backend engineer might face — e.g. weird latency issues caused by Nagle’s algorithm, or TCP CLOSE_WAIT leading to ephemeral port exhaustion between a proxy and an application node. It’s useful to understand why we mostly moved to event loop-based servers instead of thread-per-connection servers as a way to handle c10k.

A backend engineer doesn’t necessarily have to be an expert on any of these things, but they should be able to follow along if an expert explains that sort of problem.

reply

jgwil2 6 hours ago [–]

OP was responding to a comment specifically about debugging C. Someone questioned the value of this to employability and then the comment I responded to laid into him/her, suggesting no one on HN should ask that. I am defending that question. reply

cudgy 9 hours ago [–]

So much for being language agnostic reply

Diggsey 10 hours ago [–]

IME, it’s the following considerations that make back-end development hard:

• Fault tolerance. - Backwards (or forwards) compatibility. - Scalability. - Testability. - Everything around state (backup/restore, migration strategies, data integrity, etc.)

Most other things are a one-time cost. These things are an ongoing burden to consider, but if you forget to consider them it can be devastating.

Also remember: any time you give a (internal or external) customer programmatic access to something, that is an API, and APIs have huge costs to maintain. That includes when you dump your database into “data lake” for internal reporting…

reply

ianpurton 11 hours ago [–]

Some that spring to mind

• Database Migrations

• Kubernetes

• Basic RPC and code generation i.e. gRPC, OpenAPI and GraphQL.

• Realtime Concepts, i.e. Kafka, MQTT

• DevSecOps

• Builds. i.e. make files.

• Jobs, i.e. cron or batch and job workflows.

• Offsite incremental DB backups and restore.

• Infrastructure as Code i.e. Pulumi.

reply

comprev 10 hours ago [–]

• How to write clean and concise documentation, including references to further reading material which helped you solve that particular problem.
• Basics of server/runtime environment security (RBAC, least privileges, common threats, etc.)

reply

agentultra 10 hours ago [–]

• HTTP: REST/HATEOAS, headers, transport layer caching, rate limiting, load balancing

• Authentication: OAuth2 is probably the most widely used

• Authorization: RBAC

• Some rudimentary statistics: know how to read metrics, write metrics, etc

• Learn one RDBMS inside and out. Other database systems have their place but you’ll almost always encounter a Postgres, MySQL, MSSQL. Learn how to read EXPLAIN output, cursor based pagination, and indices.

reply

gilfoyle 10 hours ago [–]

Would totally recommend this book “Patterns of Enterprise Application Architecture” by Martin Fowler https://martinfowler.com/books/eaa.html

reply

Sonata 10 hours ago [–]

Having a good knowledge of HTTP is useful in many different contexts.

• The correct semantics for each HTTP method

• What different status codes indicate

• Common headers, particularly around caching

• HTTP 1.1 vs HTTP 2

• Common authentication protocols - OAuth 2.0, JWTs, etc.

reply

eatonphil 11 hours ago [–]

How to set up a decently secure, monitored and backed up server. reply

aszen 10 hours ago [–]

I have found that the most important bit as a backend engineer is modelling the application domain precisely. A taste for making good APIs helps and knowing how to measure performance. These days most backend engineers also tend to manage data sources so understanding them is also a plus

reply

yewenjie 10 hours ago [–]

Slightly related, what are the most high-quality resources for learning backend engineering? reply

koolba 10 hours ago [–]

Don’t hide the async nature of external requests. It seems like a magically good idea until it all falls apart. Timeouts and error propagation needs to flow from the request ingress all the way to the backend services. reply

bovermyer 10 hours ago [–]

Besides all of the excellent suggestions others have said, I would add:

• the OSI model, DNS, TCP/UDP, TLS, and networking in general

• CPU flame graphs and other low-level performance/debugging tools

• the Knightmare devops story

• anger management

reply

tompazourek 7 hours ago [–]

Almost everything is actually language agnostic or transferable. So it’s a big question. reply

corobo 10 hours ago [–]

Look into and learn as much as possible about penetration testing. No better way to know how to secure your code than the mindset of “Ok how would I break into this” :)

reply

etaham 11 hours ago [–]

Distributed systems, consensus & data consistency concepts reply

eterps 11 hours ago [–]

Domain Modeling (check out the book ‘Domain Modeling Made Functional’). reply

Aeolun 10 hours ago [–]

Lazy loading Dependency inversion/injection

reply

throwaway019254 11 hours ago [–]

Idempotence reply

Oleg2tor 10 hours ago [–]

REST reply

dudul 11 hours ago [–]

Isn’t an ORM the perfect example of something that is not language agnostic? reply

onei 11 hours ago [–]

The implementation is, but the concept is transferable. Pretty much the same for most concepts in programming. reply

exdsq 11 hours ago [–]

SQL is far more useful to learn than any one ORM or even the general concept of am ORM. reply

loevborg 11 hours ago [–]

ORMs are language-specific, but the insight to avoid ORMs for the rest of your life because of impedance mismatch is general and deep. reply

Apply early for the YC Winter 2022 batch

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: