Many Businesses Aren’t Protecting This Valuable Asset

full frame shot of eye

A 7-Minute Introduction to DataOps

It was a typical business morning for the Technology Consulting Group in early 2000. The Internet was booming, technology was growing in every possible genre, and we were hiring. The secretary saw the new recruit’s email come up on screen. Ten minutes later, all the documents on the network had been overwritten with zeros… the document files were there, but the content was erased.

What first looked like an email containing a resumé was actually a virus. Millions of dollars of research, invoices and contacts were gone. We had backups, of course, but Iomega Jazz drives turned out to be unreliable. Within a few months, the prosperous and growing company closed its doors. All it took was losing its intellectual property – its data.

Fast-Forward to Today

There’s a new trend across many of companies today regarding data, databases and reporting. For the past few years, companies have begun treating data, its structure, and its presentation with the same regard as software managers treat code. This is because businesses have experienced some serious hurt over the past two decades by not applying DataOps principles.

  • It costs small businesses to go bankrupt. e.g. Technology Consulting Group.
  • It costs large businesses billions of dollars in damages; e.g. Samsung, Uber, Progressive.
  • It can even make large businesses go under; e.g. It nearly ended Pixar.

What is DataOps?

DataOps is a workflow process that ensures the quality, reliability, governance and security of data, schemata, queries and reports. It offers easy delivery, quick recovery, new insights, and change transparency. It captures metadata such as the person who made the change, the associate who requested the change, what exactly was changed, tickets associated with the change, and why. It amplifies feedback loops and encourages experimentation, allowing the team to learn from mistakes to achieve mastery.

DataOps takes processes from three other well known workflows in the IT industry: Agile, DevOps and LEAN. The concept has been loosely applied in software engineering businesses since the late 2000’s, but hadn’t been formalized until mid-2021.

The Trinity of DataOps

Agile methodology is a framework for project management that that focuses on broken-down iterations of work called sprints. At the start of each sprint, a set of work items is divided amongst members of the team, or placed on a kanban board. At the end of each sprint, teams reflect on the work performed to find improvements in their strategy for the next sprint.

DevOps is the practice of team-sharing files through a central repository to coordinate and collaborate within and outside the team while communicating with each other through a set of tools. DevOps provides additional tools to consolidate work, build the product, document, unit test, systems test and, if testing is successful, deploy the product.

Lean is used to continually verify the quality of a product and the security methods that protect it. For example, definitions of data can change over time, and not having a system in place to check this allows misleading data to enter the system. Data that at one time meant “A”, now means “B” and should be handled differently. Data that could compromise the company or its clients also needs to be handled in special ways to ensure privacy.

How Does DataOps Differ From DevOps?

There are several factors that are unique to DataOps, and DataOps incorporates many aspects of DevOps within its process, but DataOps is not a superset of DevOps.

FeatureDataOpsDevOps
Sharing work on the same file?Reports are siloed and SQL scripts are functionally atomic. This makes splitting work on the same entity very difficult. Pair programming is required to share development on the same file.Source code is mapped by lines that are easily split between developers. Multiple developers can check-out work on the same file at the same time.
What teams are involved?Business Operations
Data Science
Business Intelligence
Data Governance
Data Management
Data Operations
IT Operations
Compliance
Engineering
IT Operations
Software Development
Quality Assurance
Security
User Experience
Design
Operations
What skills are involved?Data management
Data science
Data analysis
Data integration
Data quality
Data security
Statistics
Reporting
Business
IT operations
Data operations
Application engineering
Data engineering
Data governance
Requirements gathering
Application architecture
Software engineering
Software development
Application integrations
Coding
Testing
Quality control
Quality assurance
Security
IT operations
Continuous Integration
Continuous Delivery
What is the pipeline like?Develop the data product
Manage the data resources (ETL)
Test to ensure quality
Release to users
Manage usage
Monitor usage and results
Design the application / changes
Develop and build the application
Test to ensure quality
Release to users
Monitor usage and error logs
Agile PlanningUsually Kanban based; some work is planned up-front of each sprint, but most work flows through the board as requests come in. Loosely structured; More organic.Usually Scrum based; all work is planned up-front of each sprint. Highly structured, mechanical and organized.
LEANFocuses on source-of-truth and data governance principles while cards are pulled or distributed from the board throughout the Sprint.Focuses on DRY, SOLID coding principles after the Sprint has begun and work has been assigned.

What tools can be used to apply DataOps?

DevOps Source Control

TFS, Subversion, or Git can hold source files like: schemas, type table seeds, utility scripts, views, functions, procedures, configuration files, and certain types of ETL packages and reports. Basically anything that you can load in notepad and read is a good candidate for Git. However, the nature of Tableau and Power BI report files requires a little more tooling.

Power BI recently added source control to its server and it is amazing! It separates the report definition from the other components and uploads the pieces up to a Git repository. You can then compare the XML of the reports’ definitions (and other components) to see what changes took place. It provides a text box for developers to comment on version changes and commit only the reports that apply.

Tableau, however, only keeps the prior 9 copies and ditches the rest; Tableau just has a rolling backup the latest 10 versions. So for Tableau reports, either Git with LFS enabled, or Bitbucket are better options than relying on the server. Either of these options allow commits with comments, versioning, tagging, merging and conflict resolution. Of the two, I would recommend a cloud-managed Git repository system.

For data changes, such as values in type or look-up tables and system data that is handled outside an administration console, such as price changes, put the data change in a script that can be checked-in, reviewed, and vetted to a test environment. Redgate provides some good tools for this such as Flyaway and SQL Data Compare.

Recommendation: Azure DevOps Git for SQL Server & Power BI, GitHub Actions for Oracle & Tableau, Redgate Flyaway and SQL Backup Pro for data

Kanban Board

Jira, Azure DevOps, Monday and Wrike all come highly recommended. Since this is the central location for filing and working off requests and features, it’s best to research which of these would be best suited for you and your team.

Recommendation: Azure DevOps for SQL Server & Power BI, Jira or Monday for Oracle & Tableau

Lean methodology

Lean methodology rests on two pillars that provide a framework for all Lean projects: Continuous improvement and respect for people. It’s more about how to use the tools, people and resources at hand to create a feedback loop that improves process and product for the client and the workers. It can use the tools already mentioned, but adapts for the unique needs of your service. Consider a system that allows you to extend it with plugins and contains workflows and pipelines to automate as much of the development and deployment as possible.

Recommendation: Azure DevOps, Jira, Monday, Jenkins, GitLab … whatever fits your team and client needs best.

Conclusion

With tools and a structured process that involves the whole team you can implement a process that protects the core components and intellectual property of your company. It allows you to efficiently and confidently release changes that effect everyone. If there’s ever a failure because of database, report, or data changes you have a way to roll back quickly. Backups can only go so far, and should be last-ditch efforts to recover from a disaster. DataOps is a methodology that keeps your data and its availability safe and operational while keeping your teams productive.

Interviewing Your Best

Lessons learned while interviewing for a position

I’ve interviewed at Google and Amazon, and although less than 2% of the people who try for dev positions at these companies make it to these final rounds, it still hurts not getting into the 0.2% of the interviewees that land the coveted role.

Yet, following the wisdom that each failure is an opportunity to learn and improve, I’m sharing my experience and some of the lessons I’ve learned.

You’re not looking for just a job, you’re looking for a culture.

Surround yourself with an opportunity that will harness your abilities and foster growth. Look for managers who encourage growth and invest in their employees education and well-being.

Sir Richard Branson provides some of the best advice out there for company owners and employees:

“Train people well enough so they can leave, treat them well enough so they don’t want to. If You Look After Your Staff, They’ll Look After Your Customers. It’s That Simple.”

If people in the company look burned out, and they pride themselves on working 55 hours a week without rest, and the company requires salaried employees to punch in and out for breaks. Run away!

If the company follows an annual cycle of huge layoffs at the end of the year, that’s not a healthy culture.

The lesson here is to do your homework. I’ve interviewed at companies that do these things. In one case I had to stop the interview to tell the directors in the room that their business model was unhealthy and unsustainable.

Look for a company that invests in their employees as teams and as individuals.

For example, one that provides a stipend for buying course materials or encourages time off to take classes. Look for happy, enthusiastic workers. If you can interview a few employees, ask them what they’re working on and see their reaction. If it’s a sudden smile and sparkle in their eye, you know the company has some good shoulders to brush up against.

Tip: Do this at non-work holiday parties and weed out the companies where your friends roll their eyes.

If the company claims to be agile, ask them to describe the end their sprints. You want to hear words like “celebrate” and “learn” and “show off” or even “demo”. Watch out if they don’t define their “retrospective”, or if they define it in the typical ask-three-questions and move on kind-of way. That usually indicates that their “retrospective” is either flat and unfruitful – a sign that there isn’t growth – or that it’s a political finger pointing exercise.

Get business cards of everyone you meet.

If you can’t get their card, write down their names as soon as you get the chance.

This is important for two reasons:

  1. You want to address them each by name when you send them a Thank You card. More on this later.
  2. You’ll know who to connect with in the future. These are generally nice people who take an hour out of their day to help you get a job. Even if you don’t connect through LinkedIn, keep a journal of the people you contacted (receptionist included) and stay in touch with them. When needs change, you’ll be on their radar.

Don’t schedule interviews back-to-back.

This was one of the biggest mistakes of my career. Not because of lost opportunity, though that did happen, but because it was utterly disrespectful. Even a slew of apologies couldn’t mend the bad first impression that occurred when I missed an interview because a different interview scheduled before it was running late.

Now if I schedule multiple interviews on the same day, I set one in the morning and one in the late afternoon, leaving at least 3 hours between them. Otherwise, I try to keep to only one company interview a day. Some places, like Google, Amazon and Facebook take four to six hours with a series of interviews. Others, like Oracle, Cherwell or SolarWinds take about three hours with multiple segments.

Be prepared to show what you know and confess what you don’t.

Always anticipate a technical interview test. Know the difference between an Interface, an Abstract, a Protocol, and an Extension. Know a few gotcha’s about some programming languages, like how Microsoft dutifully made all their C# Math.Round functions default to Banker’s rounding. Don’t try to BS your way, and be selective of how you show off what you know.

I once suggested a certain way that C# handles immutable strings under the hood based on some random blog post I read somewhere. The blog was wrong and I looked like an idiot in front of a Microsoft engineer, who worked with the very team that developed that engine “under the hood”.

Now, unless I have first-had experience with something, I am more careful with how I introduce that bit of knowledge. Knowing your sources also helps exonerate you if your information was wrong.

If you don’t know it, admit that you have much to learn. We all do. If they insist you try, then give it your best and explain why you chose that answer.  Then after you provide your answer, ask them for the correct one.

I’ve only had one manager get stodgy and say he would never reveal answers to his interview tests … only to be immediately usurped by his best engineer with an explanation.

Be prepared to learn; you will make mistakes in the interviews.

After each interview, reflect on how it truly went. What could you do to improve and do better in the next interview? What went well in this one? Was it to know a certain tidbit of knowledge better? Was it an answer you didn’t know to one of their questions? Was it a breath mint that you forgot in the car? Did you stammer? Did you do something awkward and embarrassing?

I once tried to hold back a gaseous expulsion that released itself into the wild in such a disgusting manor that the interviewer had to call our meeting short after the first five minutes!

What can you learn? In that last case it was to not eat a delicious, spicy tex-mex burrito the day before the interview! Other answers are more subtle. Scour the web for articles on “senior engineer [coding language] interview questions and answers“.

Try an online experience-based tutorial like Interview Cake or LeetCode.com. These paid platforms aren’t cheap, but they give you insight into your abilities and what needs to be improved. Coupled with “Cracking the Coding Interview”, they can get you pretty far in the interview.

LeetCode compares the performance of your code with other submissions. Mine usually came around 85 to 95% of the most performant, and more than once I got the highest performance rating… however, simplicity and legibility sometimes suffered. It gauges various properties of your code, which is very beneficial. It even breaks questions down by the company that asks them.

Interview Cake is a bit more fun and better guided, but not as focused. If I wanted to interview for Amazon, the questions and approaches they care about are different than Facebook’s. ‘Cake provides good guides, but doesn’t suggest what company would be interested in the question you’re solving.

Always consider your audience and know your role with them. That was a difficult lesson I had to learn during an interview.

Re-ask the interview questions and re-do the interview tests at home.

Try hard to remember the interview test questions and write them down after the interview so you can revisit them later. You can also go to CareerCups.com or LeetCode.com and sometimes come across your question if you don’t remember it exactly but remember the gist of it.

The questions you were asked were chosen for several reasons. They represent a difficulty they have and gives them insight to your ability to understand and solve through that difficulty.

I’ve been asked questions in interviews and later came in to find their code is nothing like what the question presented. These are usually large non-technology corporations like insurance companies or investment firms for which I think the questions are set to be broad and generic because people shift around between departments quite a bit.

For example, you might be given a question about SQL joins (how do you create a full outer exclusive set) that you would never see or do in your career there, but knowing it demonstrates a higher level of understanding than most SQL developers out there. They’re just testing overall skill level, not problem-solving expertise or creativity.

I’ve found that following up with the interviewers with the answers you worked out at home doesn’t impress them. I though for sure that this initiative would make interviewers say to themselves something like “Hmm.. This guy doesn’t stop until he has a good answer, and he likes to work out terse problems even after it wouldn’t seem to benefit him … let’s bring him on board.”

The closest thing I ever got was a response like “Your code seemed too complex with all those fringe cases. We didn’t ask for all that.”

Contrary to what many interviewers believe, it’s a bad idea to cut ties with a non-hire as soon as the interview is done. It makes for a better society of developers and potential future employees to keep in touch. Responding thoughtfully to interviewees can be very helpful. In this example, after thinking deeper into what the prospect meant, I realized that without constraints on my requirements, I really did overthink the problem and create a bigger solution to match. That was a good lesson.

Because of that response by the interviewing HR scout, I know to ask my clients – “What are the constraints here? Time allowed to code? A Deadline? Functionality on grid ‘A’ and not Grid ‘B’? Interoperability or Extensibility through an API? Performance? etc.” That experience made me a smarter coder and more sensitive businessman.

Write thank-you letters and make them personal.

Thank you cards are vital because they show respect to the interviewer for the time they spent, and they help give you one last word as to why you are a good candidate for the position (or if you don’t want that job, it at least spreads kindness).

You want to have one sentence that specifies the time you had with each unique interviewer. Make them show appreciation and gratitude: “I appreciate the care you took in setting up the tests.” and “It was fun chatting about German shepherds! Thank you for making me feel at ease with something we have in common.”

There have been a few times that I felt terrible after an interview, but I still sent cards.

At Amazon I had an interviewer say flat out “I hate these. They’re such a waste of time.” That completely shocked me because it went against everything I understood about leadership. Amazon is all about leadership. A good leader looks forward to finding other good people to bring in their fold, even if they meet a couple of black sheep along the way.

I can’t remember the exact words I wrote on his card, but it went something like this:

Dear xxxxx,

It was a pleasure to meet you. Thank you for being so candid and honest during our interview. The time you spent with me was appreciated.

Maybe that card made him hate interviews just a little less.

Days after the interview, reflect again on how you could have improved.

Write notes. Put them in a blog. Podcast them. Get them out there to help others, and doing so helps you to remember the lessons and find some final nuggets you missed the first time. Anything prize-worthy requires many little details that make it stand out. These details come by spending more time with the customers and with the problem you’re trying to solve. Failures should always lead to learning better ways to solve the problem next time. That “ways” is plural. One mistake (other than a bad burrito) won’t cost you the entire job. Maybe along with your sloppy coding it was also a slouched posture… really dig deep – it’s called “reflection” because you have to give the water time to be still before you can see yourself.

Don’t beat yourself up.

This last bit is a post to my past and future self, but it could benefit others.

You didn’t get the position. Maybe it wasn’t a good fit. Maybe you’re skills aren’t up to par. Maybe you didn’t show the level of enthusiasm they’re looking for.

Don’t reach for that bottle of proverbial hemlock! It’s just a matter of time before you will get that position because you will learn and get better skills. You will find that perfect fit and you will be crazy-jolly when that time comes!

Why? Because if you’re reading posts like this you are also surrounding yourself with audio books, podcasts, blogs and people who have achieved the success you’re looking for. Keep it up. You’ll get there.

While you’re at it, reevaluate your life goals. What is it that really makes you happy? Is it globe trotting? Is it settling in a house with a spouse and kids? Is it creating something that changes lives? There’s a good chance you don’t need that job to achieve the deep happiness you long for.

Look for people who are open and available to coach and mentor you. Even in your current job, seek out these people and keep close to them. They are the most valuable assets to your life. When I feel incapable, mine points out what I’ve accomplished and that my intrinsic worth goes far beyond the work place. (Hi, Jeremy!) The best ones are here for you in these situations and are quick to reprimand you for being bad to yourself.

Sometimes it’s not you. It’s them.

I had one crazy interview where the interviewer asked how I used Typescript. I honestly answered that I only used it to transpile code down to ES3 for an old browser requirement. They were mortified because I didn’t use it the way they use it – to enforce strongly typed members and create (the illusion of) inheritance and immutability (they’re coding to ES5). When I asked what they were looking for, their answer was “we’re looking for someone who can solve problems creatively.”

I waited silently to see if they caught themselves in their own irony. I had just proposed a creative way to solve a problem and they balked at it, and now they’re saying that the very creativity I used is what they sought. I’ll leave you to consider why a certain big-name data provider is getting blind-sighted in the market.

Learn and practice whatever you were short on.

Back to the drawing board. What do you sense was lacking in that interview? Was it technical skills? Take an online course. Was it some factoid? Find some practice quiz online or a friend to drill you. How was your personality? Take a professional character assessment like the Myers-Briggs or DISC then reflect on how you can be more aware and presentable next time. I also suggest practicing your craft. If it’s sales, join Toastmasters. If it’s development, buy “Cracking the Coding Interview” (a book highly recommended by a friend and google employee).

When you’ve learned a new skill, passed an online exam, or assessed your natural fit. Now it’s time to do specific preparations for your next interview. After enough interviews, reviews, re-examination, failures and lessons learned you’ll conquer an interview with confidence and enjoy it.

GIT Gotchas and other things that make you cry

Lions and Tigers and Bears, Oh My!

Encountered x file(s) that should have been pointers, but weren’t

We encountered this issue when trying to merge master after someone had committed a slew of PDFs. Although we couldn’t identify the exact situation that caused this problem, the result is that nobody could merge master into their branches.

Not No Body! Not No How!
Not No Body! Not No How!

It looks like this was once filed as an issue with git, itself: https://github.com/git-lfs/git-lfs/issues/1939

But amongst the threaded comments, it looks like there is a root issue with the file types that are used to compare files and that on strange and rare occasions a unicorn fart fills the git-void with pain by changing these on the server.

After several dozen attempts, I came across this gem of a command:

$ git lfs migrate import --everything --include='*.pdf'

migrate: override changes in your working copy? [Y/n] Y
migrate: changes in your working copy will be overridden ...
migrate: Sorting commits: ..., done
migrate: Rewriting commits: 100% (5940/5940), done
migrate: Updating refs: ..., done
migrate: checkout: ..., done

The pain was almost over. Let’s see how merge does, now.

$ git merge --ff-only

fatal: Not possible to fast-forward, aborting.

Hmmm… Okay. Let’s not force a fast-forward merge; let’s just do a regular merge.

$ git merge

warning: Cannot merge binary files: Deployment Scripts/CopyFiles/UpperGreatLakesSinglesChallenge.pdf (HEAD vs. refs/remotes/origin/bugfix/13032-ems-error-saving-ppc)
:
:
CONFLICT (add/add): Merge conflict in Deployment Scripts/CopyFiles/UpperGreatLakesSinglesChallenge.pdf
Auto-merging Deployment Scripts/CopyFiles/UpperGreatLakesSinglesChallenge.pdf
:
:

Then I performed a manual merge on all the files by taking the server’s file (on the right in Sublime Merge, when resolving conflicts).
But Sublime Merge couldn’t actually commit the non-changes, so I went back to the git console to wrap up tackling the lfs-merge nightmare (like a Dream Warrior to Krueger):

$ git commit

[bugfix/13032-ems-error-saving-ppc f093b0c49] Merge remote-tracking branch 'refs/remotes/origin/bugfix/13032-ems-error-saving-ppc' into bugfix/13032-ems-error-saving-ppc
See the source image