PROJET AUTOBLOG


Mon blog-notes à moi que j'ai - Jean Baptiste Favre

Site original : Mon blog-notes à moi que j'ai - Jean Baptiste Favre

⇐ retour index

Mise à jour

Mise à jour de la base de données, veuillez patienter...

Compilation veille Twitter & RSS #2016-50

vendredi 16 décembre 2016 à 19:00

La moisson de liens pour la semaine du 12 au 16 décembre 2016. Ils ont, pour la plupart, été publiés sur mon compte Twitter. Les voici rassemblés pour ceux qui les auraient raté.

Bonne lecture

Security & Privacy

The anatomy of an IoT botnet attack
The Fastly security team is focused on leveraging our network intelligence to proactively defend the modern web. We took a look at some of the more recent (and troubling) threats on the internet, and found that the emerging IoT market is under attack. Internet-connected devices are being churned out of factories and infected by malware, or malicious code, at an alarming rate. Armies of compromised IoT devices immediately try to enroll new devices, join a botnet, and participate in large-scale DDoS attacks. As a result, we’ve recently seen some of the biggest DDoS attacks in history against journalist Brian Krebs and Dyn, launched by a massive IoT botnet of hundreds of thousands of infected devices.
Secure Desktops with Qubes: Extra Protection
This article is the fourth in my series about the Qubes operating system, a security-focused Linux distribution that compartmentalizes your common desktop tasks into individual VMs. In the previous articles, I gave a general introduction to Qubes, walked through the installation process and discussed how I personally organize my own work into different appVMs. If you haven’t read those earlier articles, I suggest you do so before diving in here. In this article, I focus on some of the more advanced security features in Qubes, including split-GPG, the usbVM and how I control where URLs open.
Privacy by design, not such a good idea
Plotted, profiled, monetized: this sums up the current situation regarding our Internet activity. The ‘free’ template has been the norm on the Internet for a few years now. Even if the services proposed by large companies such as Google and Facebook are known for offering free-seeming services, their business model is excessively profitable.
security things in Linux v4.9
Previously: v4.8.
Here are a bunch of security things I’m excited about in the newly released Linux v4.9:

System Engineering

Using Kubernetes for Deployments
Let’s learn how to set up continuous deployment to Kubernetes for your Docker apps. Specifically, we’re going to look at automating the management, deployment, and scaling of your containerized applications.
Introducing rkt’’s ability to automatically detect privilege escalation attacks on containers
Intel’s Clear Containers technology allows admins to benefit from the ease of container-based deployment without giving up the security of virtualization. For more than a year, rkt’s KVM stage1 has supported VM-based container isolation, but we can build more advanced security features atop it. Using introspection technology, we can automatically detect a wide range of privilege escalation attacks on containers and provide appropriate remediation, making it significantly more difficult for attackers to make a single compromised container the beachhead for an infrastructure-wide assault.
Kubernetes de zéro
Dans cet article, nous allons monter un cluster Kubernetes pas à pas en installant et configurant chacun des composants, sans passer par un cloud-provider all-in-one (comme GKE ou AWS) ou par un script d’installation (kube-aws, kops ou autre). Cela permet entre autres de se plonger dans le fonctionnement de chaque ressource de Kubernetes pour mieux comprendre ses mécanismes.
Containers to Clusters: Advancing Kubernetes, etcd, and more at CoreOS
At Tectonic Summit on Monday, we discussed the core premise of CoreOS: securing the internet and applying operational knowledge into software. We shared how CoreOS makes infrastructure run well and update itself automatically, from Container Linux by CoreOS, to CoreOS Tectonic – what we refer to as self-driving infrastructure.
What Kubernetes users should know about the rkt container engine
Since the release of rkt 1.0 at the beginning of this year, the project has powered ahead with over 20 new stable versions on a regular release cycle. The goal of rkt has always been to provide a container engine that is not only reliable but also composable and standards-driven, allowing easy operation and integration with other best-in-class tools in the container ecosystem. Today we wanted to provide an update on the ongoing work to integrate rkt with two such projects - the Kubernetes cluster orchestration system, and the Open Container Initiative (OCI) container standards - and chart the course for rkt’s future in the year ahead.

Monitoring

MTTD and MTTR Are Key
Mean Time To Detect (MTTD) and Mean Time To Restore (MTTR) are metrics used to describe how long it takes to discover a problem and how long it takes you to restore service relative to the start of the outage. The shorter the MTTR, the less time spent in outage and the more availability your site retains. Given that services will inevitably break at some point (Every Day is Monday in Operations), we need to be adept at restoring service as soon as possible. The service triage and restoration lifecycle is made up of several steps: detection (requiring monitoring/alerting), escalation, debugging, and remediation. Each segment of the triage needs to be measured for efficiency and effectiveness in order to keep MTTR as short as possible.
Inception: How LinkedIn Deals with Exception Logs
In early 2012, the LinkedIn Performance team was trying to build a tool to validate the health of a service after code changes (a project that led us to build EKG, our canary-monitoring system). I was assigned to look into ways to use logs to analyze a service’s health. Back then, we had a script that copied log files from different machines, ran regular expressions over them, and then provided log reports. That system worked great at the time. However, LinkedIn was growing at a very rapid rate and the script was running into scaling issues.

Software Engineering

Adding integration tests to your personal CI server
In the previous article I stepped you through the process of building a basic personal CI server with Jenkins. We got to a point where we could build a Java Web application with Gradle and deploy it to WildFly. For those that are impatient, the exported VirtualBox image with Jenkins configured using these steps is available here. The username and password combination for Linux, Jenkins and WildFly is myci and password.
Common Mistakes in Automation Testing
I deal with lots of users who use Automation Tools to test their systems every day. I interact with them for technical support, for training and consulting services. I have seen certain mistakes that are repeated more often than others. Here is a list of some of the Common Mistakes in Automation Testing.
Sanity chyecking your feature branches with a personal CI server
In the previous article I stepped you through the process of building a basic personal CI server with Jenkins. We got to a point where we could build a Java Web application with Gradle and deploy it to WildFly. For those that are impatient, the exported VirtualBox image with Jenkins configured using these steps is available here. The username and password combination for Linux, Jenkins and WildFly is myci and password.
Using best practices within the scope of automation
Introduced over the last decade and a half, automated testing has gone through many changes. The enterprise test management industry has introduced new tools, open source tools are more accessible, while quality in innovation and advancements continues to influence the market. The popularity of automated testing has even encouraged automation providers to contribute several pre-formatted frameworks designed to circumvent the requirement for in depth scripting knowledge. Industry seems to foresee a long-term relationship with automated testing.
Supporting feature branch deployments in your personal CI server
In the previous article I stepped you through the process of building a basic personal CI server with Jenkins. We got to a point where we could build a Java Web application with Gradle and deploy it to WildFly. For those that are impatient, the exported VirtualBox image with Jenkins configured using these steps is available here. The username and password combination for Linux, Jenkins and WildFly is myci and password.
How to build you own personal Jenkins CI server
In a previous article I discussed some of the benefits of running a personal CI server.

Web performances

Introducing SpeedTracker
As several reports show, it’s possible to correlate poor-performing websites with losses in engagement and revenue, so keeping a close eye on performance is of utmost importance for projects and businesses of all sizes.

Databases Engineering

MySQL & MariaDB

Row Store and Column Store Databases
In this blog post, we’ll discuss the differences between row store and column store databases.
Clients often ask us if they should or could be using columnar databases. For some applications, a columnar database is a great choice; for others, you should stick with the tried and true row-based option.
MySQL InnoDB Cluster – A Hands on Tutorial
Traditionally, setting up high availability (HA) in MySQL has been a challenging task, especially for people without advanced knowledge of MySQL. From understanding concepts and technologies, to the tooling, specific commands and files to execute and edit, there’s a lot of things you need to know even when planning a test deployment (the Quick Start Guide for Group Replication should give you an idea). So many people end up procrastinating setting up HA until disaster strikes and downtime happens.

Data Engineering & Analytics

Naive Bayes Classification explained with Python code
Machine Learning is a vast area of Computer Science that is concerned with designing algorithms which form good models of the world around us (the data coming from the world around us).

Network Engineering

Preparing for the 2016 Leap Second
Preparing for the 2016 Leap Second On 31 December this year, we’re scheduled for another leap second. There are many stories about what leap seconds can do to infrastructure and applications, and rituals are built up around them. Such rituals stem from reality: leap seconds trigger poorly-tested code paths and run contrary to assumptions that system time always runs in one direction. It’s useful to be aware of how your infrastructure handles leap seconds and how NTP servers handle them, so you can plan around the event. Here, we look at some of the NTP measurements the RIPE Atlas platform took around the last leap second, and approaches for handling them.

Management & Organization

Trained Engineers - Overnight Managers (or, The Art Of Not Destroying Your Company)
It has been said that managers shouldn’t be appointed randomly. The right people should be thoughtfully selected, should know that they’re changing their career path rather than being promoted, and should not be transitioned into management too early.

Compilation veille Twitter & RSS #2016-49

vendredi 9 décembre 2016 à 19:00

La moisson de liens pour la semaine du 5 au 9 décembre 2016. Ils ont, pour la plupart, été publiés sur mon compte Twitter. Les voici rassemblés pour ceux qui les auraient raté.

Bonne lecture

Security & Privacy

Why closing port 80 is bad for security
We’ve made some pretty big steps in our transition to a secure web but one thing that I often get asked about is closing port 80 as part of that transition. Here are my thoughts on why we shouldn’t do that.
Silence : XMPP, chiffrement et méta-données
Silence est une application libre (GPLv3) pour Android de SMS et MMS, permettant de chiffrer les communications avec les autres utilisateurs de Silence. Silence vous permet donc d’envoyer du texte et des images en tout sécurité, mais le texte et les images passeront en clair par les réseaux vers les utilisateurs classiques. Cette application est disponible sous forme de code source sur GitHub et binaire sur F-Droid et Play Store.
Progress on Privacy
The internet didn’t come with privacy, any more than the planet did. But at least the planet had nature, which provided raw materials for the privacy technologies we call clothing and shelter. On the net, we use human nature to make our own raw materials. Those include code, protocols, standards, frameworks and best practices, such as those behind free and open-source software.
Protection des données : le chiffrement ne suffit pas
Dans les protocoles de protection des données, même le plus robuste des chiffrements devient une ligne Maginot si les autres éléments du protocole sont faibles. C’est une des questions à l’ordre du jour du colloque « Sécurité informatique : mythes et réalité », organisé par le CNRS les 8 et 9 décembre à Paris.
Email Security - DMARC
The last in the email security series, DMARC, or Domain-based Message Authentication, Reporting and Conformance, builds on both SPF and DKIM. Improving security further and allowing reporting, you can monitor your domain for fraudulent or spoofed emails to take action.

System Engineering

Uber Engineering’’s Durable and Scalable Task Queue in Go
Cherami is a distributed, scalable, durable, and highly available message queue system we developed at Uber Engineering to transport asynchronous tasks. We named our task queue after a heroic carrier pigeon with the hope that this system would be just as resilient and fault-tolerant, allowing Uber’s mission-critical business logic components to depend on it for message delivery.
Secure USB boot with Debian
The moment you leave your laptop, say in a hotel room, you can no longer trust your system as it could have been modified while you were away. Think you are safe because you have a crypted disk? Well, if the boot partition is on the laptop itself, it can be manipulated and you will not notice because the boot partition can’t be encrypted. The BIOS needs to access the MBR and boot loader and that loads the Linux kernel, all uncrypted. There has been some reports lately that the Linux cryptsetup is insecure because you can spawn a root shell by hitting the enter key for 70 seconds. This is not the real threat to your system, really. If someone has physical access to your hardware, he can get a root shell in less than a second by passing init=/bin/bash as parameter to the Linux kernel in the boot loader regardless if cryptsetup is used or not! The attacker can also use other ways like booting a live system from CD/USB etc. The real insecurity here is that the uncrypted boot partition and not some script that gets executed from it. So how to prevent this physical access attack vector? Just keep reading this guide.
How we made diff pages three times faster
We serve a lot of diffs here at GitHub. Because it is computationally expensive to generate and display a diff, we’ve traditionally had to apply some very conservative limits on what gets loaded. We knew we could do better, and we set out to do so.
HTTP/2 Push: The details
HTTP/2 (h2) is here and it tastes good! One of the most interesting new features is h2 push, which allows the server to send data to the browser without having to wait for the browser to explicitly request it first.

Monitoring

Introducing Chaperone: How Uber Engineering Audits Kafka End-to-End
As Uber continues to scale, our systems generate continually more events, interservice messages, and logs. Those data needs go through Kafka to get processed. How does our platform audit all these messages in real time?

Software Engineering

Lessons in resilience at SoundCloud
Building and operating services distributed across a network is hard. Failures are inevitable. The way forward is having resiliency as a key part of design decisions.
This post talks about two key aspects of resiliency when doing RPC at scale - the circuit breaker pattern, and its power combined with client-side load balancing.
Transitioning to Python 3
The Python language, which is not new but continues to gain momentum and users as if it were, has changed remarkably little since it first was released. I don’t mean to say that Python hasn’t changed; it has grown, gaining functionality and speed, and it’s now a hot language in a variety of domains, from data science to test automation to education. But, those who last used Python 15 or 20 years ago would feel that the latest versions of the language are a natural extension and evolution of what they already know.
Securing Microservices: A Brief Look at Different Technologies
In a microservices architecture, a set of fine-grained services interact which each other to build an application or fulfill a business functionality. Each finely grained service implements a single function or a few related functions accessible over a network. This leads to an increased attack surface, making the security of a microservices architecture very important.

Databases Engineering

MySQL & MariaDB

Mysql 8.0: UUID support
In MySQL 8.0.0 we introduced many new features; among those, three new functions that ease and enhance the support for working with UUIDs.
MySQL 8.0: Storing IPv6
In MySQL 8.0.0 we introduced many new features; among those, we extended the bit-wise operations to work with binary data. Because of these changes, storing and manipulating IPv6 addresses can be done in an easier manner. In this blog we will take a look at how can you do this for some of the most common use cases.

Data Engineering & Analytics

Personalized Recommendations in LinkedIn Learning
We recently launched LinkedIn Learning, an online learning platform that enables students and professionals to take courses and learn the skills required to meet their career goals. As part of this platform, we provide personalized course recommendations. A/B testing indicates that we have 58% higher engagement rate when we provide personalized recommendations compared to generic or randomized recommendations.
Achieving a 300% speedup in ETL with Spark
A common design pattern often emerges when teams begin to stitch together existing systems and an EDH cluster: file dumps, typically in a format like CSV, are regularly uploaded to EDH, where they are then unpacked, transformed into optimal query format, and tucked away in HDFS where various EDH components can use them. When these file dumps are large or happen very often, these simple steps can significantly slow down an ingest pipeline. Part of this delay is inevitable; moving large files across the network is time-consuming because of physical limitations and can’t be readily sped up. However, the rest of the basic ingest workflow described above can often be improved.
Beginners Guide to Regression Analysis and Plot Interpretations
If you are aspiring to become a data scientist, regression is the first algorithm you need to learnmaster. Not just to clear job interviews, but to solve real world problems. Till today, a lot of consultancy firms continue to use regression techniques at a larger scale to help their clients. No doubt, it’s one of the easiest algorithms to learn, but it requires persistent effort to get to the master level.

Network Engineering

Building and scaling the Fastly network, part 2: balancing requests
The primary challenge of load balancing HTTP requests is derived from an unassailable constraint: if a packet belonging to an established TCP connection is forwarded to an incorrect server, the associated TCP flow will be reset. Unfortunately, the network layer does not understand the concept of a flow any more than applications understand the notion of packets. In the following paragraphs, we’ll outline the three traditional approaches to load balancing requests, all of which are ill-suited for a general-purpose CDN.

Management & Organization

Top 6 DevOps Metrics that Enterprise Dashboards Should Capture
Testing for websites/web applications is a constant challenge and with bubbling issues related to compatibility and security, there is a rising need for continuous development and improvement. Effective collaboration between testers and developers is becoming increasingly essential to meet the Quality Assurance (QA) goals.
Day 4 - Change Management: Keep it Simple, Stupid
I love change management. I love the confidence it gives me. I love the traceability–how it’s effectively a changelog for my environment. I love the discipline it instills in my team. If you do change management right, it allows you to move faster. But your mileage may vary.

Compilation veille Twitter & RSS #2016-48

vendredi 2 décembre 2016 à 19:00

La moisson de liens pour la semaine 28 novembre au 2 décembre 2016. Ils ont, pour la plupart, été publiés sur mon compte Twitter. Les voici rassemblés pour ceux qui les auraient raté.

Bonne lecture

Security & Privacy

Understanding CORS
RTFM… just kidding! There is no manual for the CORS (Cross-Origin Resource Sharing) specification. I really had you going there, didn’t I?
Email Security - DKIM
Domain Keys Identified Mail, or DKIM, is another security mechanism available to us that allows us to prevent spoofing or forging of emails from our domain. Using public key cryptography to assure the integrity and authenticity of emails, properly configured DKIM is an excellent protection.

System Engineering

GLB part 2: HAProxy zero-downtime, zero-delay reloads with multibinder
Recently we introduced GLB, the GitHub Load Balancer that powers GitHub.com. The GLB proxy tier, which handles TCP connection and TLS termination is powered by HAProxy, a reliable and high performance TCP and HTTP proxy daemon. As part of the design of GLB, we set out to solve a few of the common issues found when using HAProxy at scale.
Every Day Is Monday In Operations
We live in a world where our online services never sleep. Those of us who build and operate the services, however, do need to sleep—so ideally we build, monitor, alert on, and operate our services so that we can. Unfortunately, any service that is live 24/7 is in a state of change 24/7, and with change comes failures, escalations, and maybe even sleepless nights spent firefighting. Since our services must always be available, we must always be ready to answer the call. However, each problem solved is progress towards more restful nights in the future. Read on and we’ll share two war stories and lessons learned that explain why every day is Monday in operations.
Application Pauses When Running JVM Inside Linux Control Groups
Linux cgroups-based solutions (e.g., Docker, CoreOS) are increasingly being used to host multiple applications on the same host. We have been using cgroups at LinkedIn to build our own containerization product called LPS (LinkedIn Platform as a Service) and to investigate the impact of resource-limiting policies on application performance. This post presents our findings on how CPU scheduling affects the performance of Java applications in cgroups. We found that Java applications can have more and longer application pauses when using CFS (Completely Fair Scheduler) in conjunction with CFS Bandwidth Control quotas. During these pauses, the application is not responding to user requests, so this is a severe performance pain that we need to understand and address.
HPACK: the silent killer (feature) of HTTP/2
If you have experienced HTTP/2 for yourself, you are probably aware of the visible performance gains possible with HTTP/2 due to features like stream multiplexing, explicit stream dependencies, and Server Push.
There is however one important feature that is not obvious to the eye. This is the HPACK header compression. Current implementations of Apache and nginx servers, as well edge networks and CDNs using them, do not support the full HPACK implementation. We have, however, implemented the full HPACK in nginx, and upstreamed the part that performs Huffman encoding.

Software Engineering

Annotations on Document Previews
Location-specific feedback has always been fundamental to collaboration. At Dropbox, we’ve recognized this need and implemented annotations on document previews. Our goal was to allow users to provide focused and clear feedback by drawing rectangles and highlighting text on their documents. We ran into a few main challenges along the way: How do we ensure annotations can be drawn and rendered accurately on any kind of document, with any viewport size, and using any platform? How can we maintain isolation of user documents for security? How can we keep performance smooth and snappy? Below, I’m going to answer these questions and dive a bit deeper into how annotations work at Dropbox.
Asynchronous data exchanges, découpler avec classe – partie 1
Déporter des traitements lourds, transférer des logs, gérer des pics de charges, architecture réactive… Il existe de nombreux cas d’utilisation du design pattern Asynchronous data exchanges qui permet de gérer la communication de message en mode asynchrone.
On the C language, and performances
I recently gave a training to my co-workers, about the C language. Wasn’t really a training, but an introduction. With attendies really used to high level programing language, such as PHP, it was not very easy to teach some low level concepts, yet crucial to understand the power of information computation. This is mainly because we don’t cope with the same problems in low level languages, than in high.
Toggle Talk with Damian Brady
I sat down with Damian Brady, Solution Architect at Octopus Deploy for a conversation about his experience with feature toggles. He shared with me his tips for best practices, philosophies on when to flag and what he thinks the future of feature flagging will look like.
How to Manage Application Dependencies Like a Pro
As enterprises grow and scale to meet market demand, they’re finding it vital to move away from monolithic applications. Instead, a great number of organizations are transitioning to development architectures with many small components that allow them to release software much more quickly.

Web Performances

Testing with Realistic Networking Conditions
When testing performance for websites or apps that you are working on it is critical to test them with networking conditions that are representative of your users. That was one of the main reasons that I originally created WebPageTest so it was easy to test and demonstrate what performance looked like when pages were not being loaded on ultra-fast corporate networks.

Databases Engineering

DBAs, a priesthood no more
Companies have had and needed Database Administrators for years. Data is one of a business’s most important assets. That means many businesses, once they grow to the point where they must be able to rapidly scale, need someone to make sure that asset is well managed, performant for the product needs, and available to restore in case of disasters.

MySQL & MariaDB

Database Daily Ops Series: GTID Replication and Binary Logs Purge
This blog continues the ongoing series on daily operations and GTID replication.
Galera Cache (gcache) is finally recoverable on restart
This post describes how to recover Galera Cache (or gcache) on restart.
Recently Codership introduced (with Galera 3.19) a very important and long awaited feature. Now users can recover Galera cache on restart.
Using the InnoDB Buffer Pool Pre-Load Feature in MySQL 5.7
In this blog post, I’ll discuss how to use the InnoDB buffer pool pre-load feature in MySQL 5.7
Starting MySQL 5.6, you can configure MySQL to save the contents of your InnoDB buffer pool and load it on startup. Starting in MySQL 5.7, this is the default behavior. Without any special effort, MySQL saves and restores a portion of buffer pool in the default configuration. We made a similar feature available in Percona Server 5.5 – so the concept has been around for quite a while.

Data Engineering & Analytics

Artificial intelligence, revealed
It’s 8:00 am on a Tuesday morning. You’ve awoken, scanned the headlines on your phone, responded to an online post, ordered a holiday sweater for your mom, locked up the house, and are driving to work, listening to some great new music on the radio.
Difference Between Data Scientists, Data Engineers, and Software Engineers - According To LinkedIn
The differences between Data Scientists, Data Engineers, and Software engineers can get a little confusing at times. Thus, here is a guest post provided by Jake Stein, CEO at Stitch formerly RJ Metrics, which aims to clear up some of that confusion based upon LinkedIn data.

Management & Organization

Why You Need a Postmortem Process
Failure is inevitable. As engineers building and maintaining complex systems, we likely encounter failure in some form on a daily basis. Not every failure requires a postmortem, but if a failure impacts the bottom line of the business, it becomes important to follow a postmortem process. I say “follow a postmortem process” instead of “do a postmortem”, because a postmortem should have very specific goals designed to prevent future failures in your environment. Simply asking the five whys to try and determine the root cause is not enough.
Building and Motivating Engineering Teams
I have agreed to give a guest lecture for a class at Yale, and they’ve asked me to speak about “building and motivating engineering teams” from the perspective of a smaller startup. The readings for my section include A Field Guide to Software Developers by Joel Spolsky. I remember reading it when it was first written. I admire Joel’s work, and the piece has many valuable takeaways.

Compilation veille Twitter & RSS #2016-47

vendredi 25 novembre 2016 à 19:00

La moisson de liens pour la semaine du 21 au 25 novembre 2016. Ils ont, pour la plupart, été publiés sur mon compte Twitter. Les voici rassemblés pour ceux qui les auraient raté.

Bonne lecture

Security & Privacy

Internet : des clés plus fragiles qu’il n’y paraît
Des chercheurs ont démontré qu’il serait aisé de compromettre la sécurité des communications sur Internet en utilisant des nombres «truqués».
Internet est-il sûr ? Plus précisément, les protocoles qui permettent à deux ordinateurs distants de communiquer sont-ils immunisés contre le piratage ? La question est d’importance tant ces protocoles font partie de notre quotidien numérique. Une connexion sur le site de votre banque ? Celle-ci commence par la mise en place d’un canal sécurisé. Un achat en ligne ? Idem. Le paiement dématérialisé de vos impôts ? Rebelote. Or une équipe franco-américaine impliquant des chercheurs du Laboratoire lorrain de recherche en informatique et ses applications (Loria), vient de démontrer qu’il est possible de compromettre une clé de chiffrement pour la rendre quasi inopérante… sans que personne ne s’en aperçoive.

System Engineering

Kafka vs. Redis: Log Aggregation Capabilities and Performance
Today, it’s no question that we generate more logs than we ever have before. However, due to the large amount data that is constantly analyzing and resolving various issues, the process is becoming less and less straightforward.
Open Source at DigitalOcean: Introducing go-qemu and go-libvirt
At DigitalOcean, we use libvirt with QEMU to create and manage the virtual machines that compose our Droplet product. QEMU is the workhorse that enables hundreds of Droplets to run on a single server within our data centers. To perform management actions (like powering off a Droplet), we originally built automation which relied on shelling out to virsh, a command-line client used to interact with the libvirt daemon.

Monitoring

Monitoring for Black Friday: Capacity Planning Techniques
In our latest webinar, Baron Schwartz talked about how to best prepare for potential traffic spikes during Black Friday and Cyber Monday. Although some organizations may not experience a seasonal peak during the holidays, there might be other times when traffic surges can be expected–we thought we’d use the occasion of Black Friday to share some techniques to help you prepare for times when your systems may be stressed.

Software Engineering

Improve Documentation by Automating Spelling and Grammar Checks
What’s one of the first things you look at when trying a new piece of software? Or after you’ve hit that tempting Download button, what’s your usual next step? I will take a bet that for at least 70 percent of you, it’s the documentation that you check out next.
Circuit breaker, un pattern pour fiabiliser vos systèmes distribués (ou microservices) : partie 4
Nous voilà à la fin de cette série d’articles (disponibles ici, ici et ici) sur le circuit breaker.
Comment fais je pour le superviser en production ?
Notre application a passé tous les tests et il est temps de passer en production.
Setting Up and Deploying a Modern PHP Application
Despite widespread adoption, PHP has long received criticism for being inconsistent and encouraging questionable development practices. That said, millions of developers use PHP regularly, and many of them are making real software that solves real business needs, so it’s important that we understand how to use the language and deploy it in a modern environment.

Databases Engineering

Elasticsearch

Is your Elasticsearch “Trimmed”?
Here at Elastic we regularly benchmark the performance of Elasticsearch. The results are publicly available. Looking at the results, we have observed a recurring pattern of performance degradation

Cassandra

WAT - Cassandra: Row level consistency #$@&%*!
We published a blog post about some surprising and unexpected behaviors while using Apache Cassandra/DataStax Enterprise some weeks back. Recently, we encountered even more WAT moments and I believe this one is the most distressing.

Data Engineering & Analytics

Making Hard Choices: The Quest for Ethics in Machine Learning
In Silicon Valley, many companies aspire to the ideal of an ethical company. You can see this in company mottos, such as “Don’t Be Evil,” or in the social responsibility efforts espoused by many peer tech companies. On a deeper level, though, the behavior of companies like Google, Facebook, LinkedIn, and others is increasingly governed by the machine-learned systems they build to run their businesses. These companies are now starting to ask themselves how they can make an informed decision about how they operate their machine learning systems in an ethical manner, instead of being driven solely by revenue or some more abstract success metric.
A Cheat Sheet on Probability
A Cheat Sheet on Probability

Network Engineering

Impact of IPv4 Transfers on Routing Table Fragmentation
Impact of IPv4 Transfers on Routing Table Fragmentation IP address transfers have the potential to create more entries in the routing table for the same amount of address space. We analyse the net effect that four years of IPv4 transfers in the RIPE NCC service region have had on routing table growth and compare this to growth for allocations from which no transfers have been made. https://labs.ripe.net/Members/wilhelm/impact-of-ipv4-transfers-on-routing-table-fragmentation https://labs.ripe.net/logo.png

Compilation veille Twitter & RSS #2016-46

vendredi 18 novembre 2016 à 19:00

La moisson de liens pour la semaine du 14 au 18 novembre 2016. Ils ont, pour la plupart, été publiés sur mon compte Twitter. Les voici rassemblés pour ceux qui les auraient raté.

Bonne lecture

Security & Privacy

SHA-1 Certificates in Chrome
We’ve previously made several announcements about Google Chrome’s deprecation plans for SHA-1 certificates. This post provides an update on the final removal of support.
The SHA-1 cryptographic hash algorithm first showed signs of weakness over eleven years ago and recent research points to the imminent possibility of attacks that could directly impact the integrity of the Web PKI. To protect users from such attacks, Chrome will stop trusting certificates that use the SHA-1 algorithm, and visiting a site using such a certificate will result in an interstitial warning.
Mission Improbable: Hardening Android for Security And Privacy
After a long wait, the Tor project is happy to announce a refresh of our Tor-enabled Android phone prototype.
This prototype is meant to show a possible direction for Tor on mobile. While I use it myself for my personal communications, it has some rough edges, and installation and update will require familiarity with Linux.

System Engineering

Visualize Kubelet Performance with Node Dashboard
In Kubernetes 1.4, we introduced a new node performance analysis tool, called the node performance dashboard, to visualize and explore the behavior of the Kubelet in much richer details. This new feature will make it easy to understand and improve code performance for Kubelet developers, and lets cluster maintainer to decide configurations according to provided Service Level Objectives (SLOs).
LinkedIn’s Next-Generation Data Center Goes Live
Earlier this year we announced Project Altair, our massively scalable, next-generation data center design. We also announced our plans to build a new data center in Oregon, in order to be able to more reliably deliver our services to our members and customers. Today, we’d like to announce that our Oregon data center, featuring the design innovations of Project Altair, is fully live and ramped. The primary criteria when selecting the Oregon location were: procuring a direct access contract for 100% renewable energy, network diversity, expansion capabilities, and talent opportunities.
Infrastructure Update: Pushing the edges of our global performance
Dropbox has hundreds of millions of registered users, and we’re always hard at work to ensure our customers have a speedy, reliable experience, wherever they are. Today, I am excited to announce an expansion to our global infrastructure that will deliver faster transfer speeds and improved performance for our customers around the world.
How to solve anything in VCL, part 3: authentication and feature flags at the edge
In “How to solve anything” parts 1 and 2, we outlined how to use Varnish Configuration Language (VCL) to address some of your more challenging problems. In this post, we’ll discuss how Andrew Betts of the Financial Times uses advanced VCL to securely cache and serve authenticated and authorized content, and set up feature flags.
Performance Tuning HAProxy
In a recent article, I covered how to tune the NGINX webserver for a simple static HTML page. In this article, we are going to once again explore those performance-tuning concepts and walk through some basic tuning options for HAProxy.
How Urban Airship Scaled to 2.5 Billion Notifications During the U.S. Election
Urban Airship is trusted by thousands of businesses looking to grow with mobile. Urban Airship is a seven year old SaaS company and has a freemium business model so you can try it for free. For more information, visit www.urbanairship.com. Urban Airship now averages more than one billion push notifications delivered daily. This post highlights Urban Airship notification usage for the 2016 U.S. election, exploring the architecture of the system–the Core Delivery Pipeline–that delivers billions of real-time notifications for news publishers.

Monitoring

Monitoring items for uneven values, how odd is that?
As someone working in IT infrastructure, every now and then you are confronted with a problem that you are not certain how to solve. Often times I have found myself overthinking things and ending up with a complex solution that isn’t very elegant but get’s the job done.

Software Engineering

Circuit breaker, un pattern pour fiabiliser vos systèmes distribués (ou microservices) : partie 3
Maintenant que nous avons vu la théorie sur les précédents articles disponibles ici et ici, penchons-nous sur la pratique.
HTTP/2 : la transition est en marche ! Quels changements pour le développeur front-end ?
Nous l’avons annoncé il y a quelques semaines : notre outil de test de performance est maintenant pleinement compatible HTTP/2 et notre référentiel de bonnes pratiques de performance a été adapté pour prendre en compte les particularités de ce protocole. Je vous propose aujourd’hui de revenir en détail sur ce qui a motivé la naissance de HTTP/2, les changements majeurs apportés, mais aussi les différentes bonnes pratiques HTTP/1 dont nous allons devoir nous défaire !
Engineering Infrastructure at Scale: Test Tracking
This blog series describes the engineering infrastructure (technologies, processes, tools, and culture) that enables several hundred engineers across LinkedIn to innovate and release software continuously with agility, quality, and productivity. This post describes the analytics infrastructure across iOS, Android, web, and API.
In order to collect information on how members interact with LinkedIn apps and websites, we’ve built a powerful tracking infrastructure framework at LinkedIn. This framework has allowed us to assess whether new features are successful, to conduct business auditing, and to gain insight into member behavior in general. What we call “tracking” is also referred to as “metrics” or “analytics” at some companies.
Dans le Libre : la bifurcation (fork)
Dans cette série d’articles consacrés aux usages du Libre et après avoir abordé les principales étapes de la démarche du Libriste pour résoudre un problème (gratter ses propres démangeaisons) par la création puis l’utilisation d’un programme (manger ce que l’on prépare) puis enfin l’automatisation de sa conception et de son déploiement (tout automatiser), nous nous intéresserons à l’une des pratiques les controversées mais aussi des plus efficaces du Logiciel Libre : la bifurcation (fork en anglais).

Databases Engineering

Elasticsearch

A New Way To Ingest - Part 2
This is the second part of a two-part series about ingest nodes, a new feature in Elasticsearch 5.0.
In the first part we talked about what ingest nodes are, and how to configure and use them. In this second part we will focus on how to use ingest nodes as part of a deployment of the Elastic Stack.

MySQL & MariaDB

All You Need to Know About GCache (Galera-Cache)
Percona XtraDB Cluster is a multi-master topology, where a transaction executed on one node is replicated on another node(s) of the cluster. This transaction is then copied over from the group channel to Galera-Cache followed by apply action.
The cache can be discarded immediately once the transaction is applied, but retaining it can help promote a node as a DONOR node serving write-sets for a newly booted node.
So in short, GCache acts as a temporary storage for replicated transactions.
Scaling MySQL with TCP Load Balancing and Galera Cluster
We introduced TCP load balancing in NGINX Plus R5, and have continually added features in subsequent releases, as well as support for UDP load balancing. In this article we explore the key requirements for TCP load balancing and how NGINX Plus addresses them.
To explore the features of NGINX Plus we will use a simple test environment that represents the key components of an application with a scaled database backend. For complete instructions on building the test environment, see Appendix 1.

Data Engineering & Analytics

Open-Sourcing Yelp’s Data Pipeline
For the past few months we’ve been spreading the word about our shiny new Data Pipeline: a Python-based tool that streams and transforms real-time data to services that need it. We wrote a series of blog posts covering how we replicate messages from our MySQL tables, how we track schemas and compute schema migrations, and finally how we connect our data to different types of data targets like Redshift and Salesforce.
The Story of Batching to Streaming Analytics at Optimizely
Our mission at Optimizely is to help decision makers turn data into action. This requires us to move data with speed and reliability. We track billions of user events, such as page views, clicks and custom events, on a daily basis. To provide our customers with immediate access to key business insights about their users has always been our top most priority. Because of this, we are constantly innovating on our data ingestion pipeline.
Regression (LR and MLR) and differences, not for the Economy. Professional analyst should be able to answer these three questions.
To produce a regression analysis of inference that can be justified or trustworthy in the sense that helpful. The term in the statistical methods that generate a linear the best estimator is not bias (best linear unbiased estimator) abbreviated BLUE. Then there are some other things that are also important to note, in which the data to be processed, must meet certain requirements. In terms of statistical methods some terms or conditions of the so-called classical assumption test. Because they meet the assumptions of classical statistical coefficient will be obtained which actually became estimator of parameters that can be justified or accurate, among others:

Management & Organization

Etsy’s Debriefing Facilitation Guide for Blameless Postmortems
In 2012, I wrote a post for the Code As Craft blog about how we approach learning from accidents and mistakes at Etsy. I wrote about the perspectives and concepts behind what is known (in the world of Systems Safety and Human Factors) as the New View on “human error.” I also wrote about what it means for an organization to take a different approach, philosophically, to learn from accidents, and that Etsy was such an organization.