The gate to Latin America with a population of 200M people. It's no surprise that a country of this size, with a rapidly growing Internet user-base, faces many challenges. If you're based in Brazil and developing an app to address these challenges, then let’s work together! Join us in São Paulo for six months to accelerate your app!
The Automatic app connects to the Automatic car adapter over Bluetooth and displays the details of every trip. The app offers features that every driver should have, like engine code diagnostics, emergency crash response from a live representative, a web app for desktop where you can review your data, and extensions for Apple Watch and Pebble.
DuckDuckGo, the privacy-focused web search engine startup, continues to grow rapidly. But it has also struggled to crack the mainstream.
Through Dec. 15, DuckDuckGo received 3.25 billion search queries this year, according to its publicly posted traffic statistics. That’s up 74% over the same period last year. Monday, Dec. 14 was its first day with more than 12 million direct queries.
But relative to search industry leader Google, DuckDuckGo is still very small. Google has boasted more than 100 billion monthly search queries for a while, whereas DuckDuckGo’s direct queries passed 325 million last month. (DuckDuckGo also processed another 284 million queries in November via its API and from bots.)
Another data point: DuckDuckGo drew 3 million unique visitors on desktop computers in October, according to comScore. That represents more than 100% year-over-year growth, but does not indicate a very large audience. (It’s worth noting the possibility that users of a privacy-focused search engine also make it difficult for comScore to track them.) DuckDuckGo doesn’t disclose what percentage of its searches are conducted on mobile devices, but Google said this year more than half of its searches are now mobile.
What’s driving growth? DuckDuckGo CEO Gabriel Weinberg, reached via email, credits partnerships launched in 2014 with Apple and Mozilla, and word of mouth.
He also passes along a Pew study from earlier this year, where 40% of American respondents said they thought search engines “shouldn’t retain information about their activity.” (Google and others use information, including your search queries, to build profiles for ad targeting, among other purposes.)
Yet that hasn’t translated to action. If 40% of US search queries went through DuckDuckGo, it would be more than half Google’s current size. It seems driving awareness should be a priority for 2016.
“Our biggest challenge is that most people have not heard of us,” Weinberg says. “We very much want to break out into the mainstream.”
On the night of 7 January 2012, a stationmaster at a provincial railway station in central India discovered the body of a young woman lying beside the tracks. The corpse, clothed in a red kurta and a violet and grey Puma jacket, was taken to a local morgue, where a postmortem report classified the death as a homicide.
The unidentified body was “a female aged about 21 to 25 years”, according to the postmortem, which described “dried blood present” in the nostrils, and the “tongue found clenched between upper and lower jaw, two upper teeth found missing, lips found bruised”. There was a crescent of scratches on the young woman’s face, as if gouged by the fingernails of a hand forcefully clamped over her mouth. “In our opinion,” the handwritten report concluded, “[the] deceased died of asphyxia (violent asphyxia) as a result of smothering.”
Three weeks later, a retired schoolteacher, Mehtab Singh Damor, identified the body as his 19-year-old daughter Namrata Damor – who had been studying medicine at the Mahatma Gandhi Medical College in Indore before she suddenly vanished one morning in early January 2012. Damor demanded an investigation to find his daughter’s killer, but the police dismissed the findings of the initial postmortem, and labelled her death a suicide.
The case was closed – until this July, more than three years later, when a 38-year-old television reporter named Akshay Singh travelled from Delhi to the small Madhya Pradesh town of Meghnagar to interview Namrata’s father. Singh thought that Namrata’s mysterious death might be connected to an extraordinary public scandal, known as the Vyapam scam, which had roiled the highest echelons of the government of Madhya Pradesh.
For at least five years, thousands of young men and women had paid bribes worth millions of pounds in total to a network of fixers and political operatives to rig the official examinations run by the Madhya Pradesh Vyavsayik Pariksha Mandal – known as Vyapam – a state body that conducted standardised tests for thousands of highly coveted government jobs and admissions to state-run medical colleges. When the scandal first came to light in 2013, it threatened to paralyse the entire machinery of the state administration: thousands of jobs appeared to have been obtained by fraudulent means, medical schools were tainted by the spectre of corrupt admissions, and dozens of officials were implicated in helping friends and relatives to cheat the exams.
A fevered investigation began, and hundreds of arrests were made
A fevered investigation began, and hundreds of arrests were made. But Singh suspected that the unsolved murder of Namrata Damor – and the baffling insistence of the police that she had flung herself from a moving train – might be part of a massive cover-up, intended to protect senior political figures, all the way up to the powerful chief minister of Madhya Pradesh.
By the time Singh came to interview Mehtab Singh Damor, the Vyapam scam had begun to seem like something more deadly than an unusually large bribery scandal. Since 2010, more than 40 doctors, medical students, policemen and civil servants with links to the Vyapam scam had died in mysterious circumstances.
The state government, under relentless pressure from its political opponents, insisted that none of the dead had been murdered – the media, they contended, had simply stitched together a series of unconnected natural deaths. “Whoever is born has to die one day,” the state’s home minister, Babulal Gaur, said in a memorable TV interview, citing Hindu scripture. “This is the mrityu lok” – the realm of death.
When Akshay Singh arrived at the Damor home, Namrata’s father sat him down on a heavy wooden armchair in his living room, silenced the television, and handed over a well-thumbed file stuffed with photocopies of petitions, police reports, and court papers. Tea arrived on a tray; Singh picked up a cup and turned his attention to Namrata’s postmortem report and the coroner’s feathery scrawl.
As he sipped his tea, Singh turned as if to ask a question, but his face froze, his left arm shivered, his open mouth gasped for air, tiny bubbles of spittle formed on his lips before he collapsed in his chair, the dead woman’s case file motionless in his lap.
“We lay him down on the floor, loosened his clothes and sprinkled his face with water,” recalled Rahul Kariaya, an Indore-based journalist who took Singh to Damor’s house, “I checked his pulse and I knew right away, Akshay Singh was dead.”
Within hours of Singh’s death, the long-simmering Vyapam scandal exploded from the inside pages of newspapers on to primetime television. It soon emerged that the Madhya Pradesh police had ended their investigation into Namrata Damor’s death on the basis of a second postmortem report, prepared by a doctor who later admitted that he had not examined the body and had based his findings solely on photographs provided by the police. Namrata Damor, according to this postmortem, had killed herself because of a failed relationship.
“I want everyone across the country to ask themselves one question,” the country’s most bombastic TV news presenter, Arnab Goswami, bellowed one night in July, waving copies of both postmortem reports on his popular nightly programme. “How does a postmortem come to the conclusion, [without] using any evidence, that the ‘victim is disappointed in love and has caused annoyance of her parents?’”
The Vyapam scandal began as an old-fashioned scam in a country with a long and storied history of corruption. Officials at the testing agency, along with their political backers and a network of fixers and touts, had charged preposterous sums of money to guarantee candidates either a government job or admission to a state medical college by fixing the results of the entrance examinations. Cheating of this sort was not a new phenomenon – but the enormous scale of the racket, the involvement of top government officials and medical colleges, and the alleged murder of suspects made the Vyapam scam into an explosive political controversy. The four-week long summer session of the Indian parliament in 2015 was completely paralysed by demands from the opposition Congress party for the resignation of Shivraj Singh Chouhan, the Bharatiya Janata Party (BJP) chief minister of Madhya Pradesh.
When the scandal first became public in 2013, the Madhya Pradesh government assembled a special taskforce of state police officers to investigate the allegations. It moved quickly, sweeping up everyone from individual students and their parents to the chief minister’s personal secretary. By the time of Singh’s sudden death in July 2015, the taskforce had already arrested an astonishing 2,235 people – of whom 1,860 were released on bail after questioning.
The list of top state officials placed under arrest reads like the telephone directory of the Madhya Pradesh secretariat. The most senior minister in the state government, Laxmikant Sharma – who had held the health, education and mining portfolios – was jailed, and remains in custody, along with his former aide, Sudhir Sharma, a former schoolteacher who parlayed his political connections into a vast mining fortune. Another minister, Gulab Singh Kirar, simply disappeared rather than face questioning from the police. (Many of those accused have protested their innocence, but it may take years for the prosecution to secure any convictions.)
Among those detained by the taskforce were half a dozen aides to top state ministers – but from the start, opposition parties insisted that the probe was an elaborate charade, intended to convey a sense of urgency to the public while protecting the chief minister. The preponderance of aides among the arrested fuelled speculation that underlings had been forced to fall on their swords to protect their bosses.
And then, as the investigation widened, people started dying. Some had perished before the taskforce had a chance to interrogate them – such as Anuj Uieke, a medical student accused of working as a middleman connecting exam aspirants and Vyapam officials. He died along with two friends also accused of involvement in the scam when a truck ploughed into their car in 2010. Others apparently took their own lives, like Dr Ramendra Singh Bhadouriya, who was accused of cheating his way to a medical college seat in 2008 and then helping others do the same. He was found hanging from the ceiling fan in his home in January 2015. (Five days later, his mother took her own life by drinking acid.) Another suspect, Narendra Tomar – a seemingly healthy 29-year-old veterinary doctor at a government hospital, who had been arrested for his role as a middleman in the scam – had a sudden heart attack in jail this June and died in hospital the next day.
In July 2014, the dean of a medical college in Jabalpur, Madhya Pradesh, Dr SK Sakalle – who was not implicated in the scandal, but had reportedly investigated fraudulent medical admissions and expelled students accused of obtaining their seats by cheating – was found burned to death on the front lawn of his own home. The police initially maintained that Sakelle had doused himself in kerosene and set himself alight; an unusual means of suicide for a doctor with easy access to a wide range of toxins. But they were forced to reopen their investigation one year later, when Sakelle’s colleague and close friend, Dr Arun Sharma, who took over as dean of the medical college, was found dead in a Delhi hotel alongside a half-empty bottle of whisky and a strip of anti-depressant pills.
In March of this year, Shailesh Yadav, the 50-year-old son of the governor of Madhya Pradesh – the formal head of state government, appointed by the president of India – died of a suspected brain haemorrhage in his family home. Both the governor and his son were implicated in the scam.
Each new death brought a flurry of headlines, and increasingly excited speculation about conspiracies and cover-ups. “Who is killing all these people?” the TV presenter Goswami demanded on air one night – inviting viewers to tag their tweets on the story with #KillingAScam.
Through it all, the government of Madhya Pradesh insisted that the series of deaths was nothing more than a coincidence – a conspiracy theory cooked up by its political opponents – and that the state police taskforce was conducting an exemplary investigation of the scandal. But the death of Akshay Singh, which brought the national spotlight on to the case, forced the government to ask for a probe by the notionally autonomous Central Bureau of Investigation (CBI). In an interview with the Indian Express newspaper, Chouhan, the chief minister of Madhya Pradesh, said he was satisfied by the state police investigation, but agreed to a CBI investigation to clear “an atmosphere of falsehood” that “would have eventually affected the wellbeing of the state”.
When I arrived in the state capital, Bhopal, a beautiful city of lakes, greenery, and double-storey buildings, at the end of July, the local press had coined a cruel pun on the chief minister’s first name. Rather than Shivraj – after the Hindu god Shiva – they were calling him Shavraj, or King of the Corpses.
In 2013, the year the scam was first revealed, two million young people in Madhya Pradesh – a state the size of Poland, with a population greater than the UK – sat for 27 different examinations conducted by Vyapam. Many of these exams are intensely competitive. In 2013, the prestigious Pre-Medical Test (PMT), which determines admission to medical school, had 40,086 applicants competing for just 1,659 seats; the unfortunately named Drug Inspector Recruitment Test (DIRT), had 9,982 candidates striving for 16 vacancies in the state department of public health.
For most applicants, the likelihood of attaining even low-ranking government jobs, with their promise of long-term employment and state pensions, is incredibly remote. In 2013, almost 450,000 young men and women took the exam to become one of the 7,276 police constables recruited that year – a post with a starting salary of 9,300 rupees (£91) per month. Another 270,000 appeared for the recruitment examination to fill slightly more than 2,000 positions at the lowest rank in the state forest service.
It was on the morning of the medical exam in 2013 that the Vyapam scandal began to unravel. A team of policemen raided the Hotel Pathik, a seedy £5-a-night motel on the outskirts of Indore, the largest city in Madhya Pradesh.
In room 13, the police came upon a young man readying himself for that morning’s exam. He handed over a voter identity card, introducing himself as exam candidate Rishikesh Tyagi, but when the police asked him his father’s name and his date of birth, he said he could not remember.
“On doing strict interrogation,” a police report of the incident reads, “he told his correct name as Ramashankar … he told us he came to give the examination in the name of Rishikesh Tyagi.”
Ramashankar, the police alleged, was already studying medicine in Uttar Pradesh, and had accepted 50,000 rupees (£500) to take the exam on behalf of Rishikesh Tyagi. Twenty such impostors were arrested that morning, 18 of whom had come from out of town to impersonate young students who felt they could not pass the entrance exams themselves.
The impersonators led the police to Jagdish Sagar, a crooked Indore doctor who had set up a lucrative business that charged up to 200,000 rupees (£2,000) to arrange for intelligent but financially needy medical students to sit examinations on behalf of applicants who could afford to pay. Police claimed that Sagar had amassed a fortune in land, luxury cars, and jewellery from the racket: according to a report in the Hindustan Times (headline: “Vintage Wine, Bed of Cash”), he slept on a mattress stuffed with 1,300,000 rupees. But Sagar, one policeman told me, was only the most prominent of a swarm of middlemen who offered similar services.
Standardised testing in India is a heroic and misguided attempt to compensate, over three short hours, for a young lifetime’s worth of inequities of caste, class, gender, language, region and religion, and the crushing inadequacy of the state-run schooling system. It is the only consideration for achieving college admissions or government employment. Nothing else matters – not your grades over 12 years of school, nor any hobbies, interests or transformative life experiences.
The competition is so intense, and India’s schools so poor that, according to the National Statistics and Sampling Office, a quarter of all students in India are enrolled in tuition centres. In some states, that figure is as high as 90%. The private tuition industry grew 35% over a five-year period from 2008 to 2013, as reported in a 2013 survey by the Indian Associated Chambers of Commerce and Industry, and is expected to be worth around £27bn by 2015.
This fevered demand for after-school classes has turned tuition centres into well-known brands – represented by star students who have secured the highest test results, whose bespectacled and slightly woebegone faces are plastered, like football stars, on billboards along rural highways, crowded railway stations, dusty bus stands, and outside schools and colleges.
The tuition industry has taken over entire towns, such as Kanpur in Uttar Pradesh, where students come from across the country to sleep in cramped hostels, subsisting on fried snacks and shuttling from coaching centre to coaching centre under the tutelage of “celebrity” teachers. While established centres advertise “world-class” facilities and faculties, less well-known institutes offer tales of improbable success, such as the poster I saw in Kanpur for a centre called The New Tech Education, featuring four young women in hijab and celebrating a “Miracle in the history of pre medical institute, all the 4 sisters of middle class family became doctor.”
The explosion of tuition centres, and the scarcity of jobs, has only intensified the desperation to grasp the tiny number of university places and jobs that are made available each year. “The impostor system has its roots in the world of the tuition centres,” I was told by a medical student who I met at his college hostel in Gorakhpur, in the neighbouring state of Uttar Pradesh, where Sagar had recruited most of his fraudulent candidates. “Centres conduct weekly exams and post the results on their bulletin boards, which makes it easy for touts to spot both – bright students who can work as impostors, and weak students looking to cheat their tests,” he said.
Once I started doing well in my tests, touts started calling, promising me 100,000 rupees for two days’ work
“Once I started doing well in my tests, touts started calling me every week promising me 100,000 rupees (£1,000) for two days’ work. He said, I’d be flown to an examination centre, put up in a five-star hotel, and flown back to home – all I had to do was solve a paper.” My source said he had declined their overtures – but one of his classmates did not, and is now in jail in a Vyapam-related case.
Beginning in the late 1990s, investigators allege, Sagar and his out-of-town impostors helped hundreds of students cheat the medical exams. But eventually Sagar’s ambitions widened, and he turned to a man called Nitin Mohindra.
Nitin Mohindra is a short, pudgy man with a receding hairline, descending paunch, and lampshade moustache, who joined Vyapam in 1986 at the age of 21 as a data entry operator. By the time he met Jagdish Sagar, he had risen through the ranks to become the agency’s principal systems analyst, without ever drawing much attention to himself.
His colleagues recalled that he had only twice been the subject of any office gossip – both times for arriving at work in slightly flashy new cars: a Honda City in 2008, and Renault Duster SUV a few years later. “And he wore very nice shirts,” one colleague told me. “Nothing fancy, but you could tell that the material was just better quality than everyone else’s shirts.”
In 2009, police claim, Sagar and Mohindra had a meeting in Sagar’s car in Bhopal’s New Market bazaar, where the doctor made an unusual proposition: he would give Mohindra the application forms of groups of test-takers, and Mohindra would alter their “roll numbers” to ensure they were seated together so they could cheat from each other. According to Mohindra’s statement to the police, Sagar “offered to pay me 25,000 rupees (£250) for each roll number I changed.”
This came to be known as the “engine-bogie” system. The “engine” would be one of Sagar’s impostors – a bright student from a medical college, taking the exam on behalf of a paying customer – who would also pull along the lower-paying clients sitting next to him by supplying them with answers.
Vyapam officials showed me seating plans from examination centres to illustrate how Mohindra ensured that engines and bogies not only sat together, but were allotted seats in the last few benches in each examination room – far from the moderator – to make it easier for them to cheat. From 2009 to 2013, the police claim, Mohindra tampered with seating assignment for at least 737 of Sagar’s clients taking the state medical exam.
The scam became even more sophisticated in 2011, when the Madhya Pradesh state government appointed Pankaj Trivedi, a lecturer at a government college in Indore, as the controller of examinations at Vyapam – responsible for ensuring the security of the testing process. According to the police, Trivedi, who is now in jail, came under pressure from influential state ministers and officials to provide jobs and admissions for their relatives and friends. New to the job, Trivedi turned to Mohindra, who devised a solution that was dazzling in its simplicity. Students who had paid to have their results fixed were told to attempt only those questions for which they knew the answers, and leave the rest blank.
“Mohindra hooked all of Vyapam’s computers to a common office network and retained all administrator privileges,” said Tarun Pithode, an energetic young civil servant who was appointed Vyapam’s new director to set things straight after the scam broke. After the multiple-choice exam sheets were scanned, Mohindra could access the computer that stored the results, and alter the answers as he wished.
Once the results had been altered on the computer, Mohindra would approach the exam observers and ask for the original answer sheet, claiming that the student had requested a copy under India’s Right to Information Act. He would then sit in Trivedi’s office and fill out the originals so that they tallied with the altered version saved on the computer
The most glaring example of this method was discovered in the case of Anita Prasad, a daughter of Prem Chand Prasad, the personal secretary to Chief Minister Chouhan. She had passed the PMT exam in 2012 with the assistance of Trivedi and Mohindra – but failed to follow their instructions, and attempted to answer all of the questions, many of them incorrectly. When police investigators later obtained a copy of her original answer sheet, they found that Vyapam officials had used correction fluid to blank out her wrong answers and pencil in the correct ones instead.
“In our review, we found almost every system had been subverted,” Pithode said. “For example, every question paper set has an ‘answer key’ that is put into a self-sealing envelope before the exam and opened only at the time of tabulating results. Trivedi would seal the envelope in the presence of observers, but later would simply tear open the envelope, make copies of the key, and put the original document into a new envelope.”
Over the course of only two years, police allege, Mohindra and Trivedi conspired to fix the results of 13 different examinations – for doctors, food inspectors, transport constables, police constables and police sub-inspectors, two different kinds of school teachers, dairy supply officers and forest guards – which had been taken by a total of 3.2 million students.
Amidst this tangle of impostors, engine-bogies and altered answer sheets, the police soon found a spreadsheet on Nitin Mohindra’s hard drive that listed the names of hundreds of students who had paid to cheat the exams – along with the names of the minister, bureaucrat or fixer who had referred the student to Mohindra and the agreed payment.
The list included political heavyweights such as Uma Bharati, a former chief minister of Madhya Pradesh who is now the water minister in Prime Minister Narendra Modi’s cabinet – and a longtime rival of Shivraj Singh Chouhan.
With the police in possession of Mohindra’s spreadsheet and a ready list of suspects, the case seemed to be heading to a speedy closure. But in February this year, the opposition Congress party – which had been demanding for months that Chouhan resign over the scam – made a startling claim. The police taskforce, they alleged, had tampered with the spreadsheet of conspirators to remove the name of the chief minister and replace it with the names of his rivals. The changes were made in “at least 48 places”, the Congress leader Digvijaya Singh – another former chief minister of Madhya Pradesh – claimed when we met in his office in New Delhi. “The chief minister’s name was either removed all together, or replaced with the name of Uma Bharati.”
The Central Bureau of Investigation took over the case in July of this year. But there is little reason to believe that its findings will resolve the scandal. In 2013, India’s Supreme Court famously described the country’s premier investigating agency as a “caged parrot” for its susceptibility to political pressure from the reigning central government. And while the investigation drags on, it has been further muddied by an elaborate and increasingly impenetrable series of allegations and counter-allegations between the ruling BJP and the opposition Congress over the veracity of the evidence seized from Mohindra’s computer.
Two days after Mohindra’s arrest in 2013, a policeman showed up at the office of Prashant Pandey, a cyber-security expert with pointed sideburns, bouffant hair, a handlebar moustache and a taste for fitted waistcoats.
Pandey is the proprietor of a firm called Techn07. In a detailed resume he sent me after our first meeting, Pandey claimed to have helped the Madhya Pradesh police crack cases involving Islamist terrorists, tax evaders, kidnappings, and political murders – along with the Vyapam investigation. (The head of the Vyapam taskforce did not respond to multiple requests for comment, but state officials and police officers confirmed that Pandey had done work for the Indian Revenue Service and the Vyapam taskforce.)
“The policeman said his seniors had seized a hard drive from a suspect, but the local police station did not have a SATA cable to connect it to their desktop,” Pandey recalled when we met in his lawyer’s office in Delhi. “I gave him a cable, but the policeman wanted to make sure the connector was working and so he plugged it into one of my computers.” Pandey said that once he connected the drive, his computer automatically began to make a mirror image. The policeman had a cup of tea and left.
Two months later, Pandey said, the Vyapam taskforce paid him to install hidden cameras in their interrogation cells in Bhopal. (Pandey showed me a bill, dated March 2014, for a “bullet camera”, a Sony voice recorder, and various accessories, along with copies of cheques from the state finance ministry, to prove he had worked for the government in the past.)
Pandey said that he had worked for the Vyapam taskforce for more than six months – until the relationship soured in June 2014, when the Congress party released phone records alleging that Sadhna Singh, the wife of Shivraj Singh Chouhan, had made 139 calls from the chief minister’s residence to the ringleaders of the scam, Nitin Mohindra and Pankaj Trivedi. (The chief minister dismissed the allegations as a fabrication, and the matter was never investigated by the police.)
The police arrested Pandey, and jailed him on charges of trying to sell confidential phone records. His computers were seized, he said, and he was interrogated for three days. “All they asked me was, ‘What do you know about the chief minister and Vyapam.’” He was released on bail, but claims he was picked up again in the middle of the night, and taken to a safe house for further interrogation. “They said, you give anyone any more information and you are finished,” he told me.
I decided to fight back and expose all these corrupt officials. I realised that the police had doctored the evidence
“I decided to fight back and expose all these corrupt officials,” Pandey continued. “I realised that I had a mirror image of Nitin Mohindra’s hard drive, and on comparing the Excel sheet submitted by the police in court, and the Excel sheet from my copy of Mohindra’s hard drive, I realised that the police had doctored the evidence to save the chief minister.” Through his lawyer, Pandey leaked the information to the Congress leader Digvijaya Singh, who released it to the public.
The state government insists that it is Pandey – and the Congress party – who have tampered with the evidence. The Madhya Pradesh police submitted Mohindra’s hard drive to a government forensic laboratory in Gujarat, which certified the authenticity of their version of the document. Pandey’s lawyers, on the other hand, submitted his copy of the spreadsheet to a well-regarded private forensic lab in Bangalore, which verified that his was the original copy. So the BJP-led government has its own version of the evidence, and the Congress opposition has another – a neat parable for the general condition of Indian political debate. It will fall to the Supreme Court (and perhaps yet another forensic lab) to decide whose report to believe.
In the meantime, Mohindra’s lawyer, Ehtesham Qureshi, has seized on the controversy to insist that the case against his client has been fabricated. “If the police have altered the Excel sheet in my client’s hard drive,” Qureshi told me, “how can we trust any of the supposed evidence seized from his computer?” The police, he alleges, have kept Mohindra in jail for two years by filing a succession of cases against him, one each 90 days, to prevent him from becoming eligible for bail. “My client, who is a poor, middle-class person, is being made a scapegoat to protect the wealthy and powerful,” Qureshi said. “They are going to keep him in forever, because if he gets out and starts speaking, who knows what will happen.”
When I met Rajendra Shukla, Madhya Pradesh’s minister for public relations, power and electricity, and mines and minerals – the man with the unenviable task of managing the fallout from the Vyapam scam – he calmly insisted that the massive scale of the multiple ongoing investigations was proof that the state government had nothing to hide.
“Several senior people have been arrested in connection with this case,” Shukla told me one evening at his residence in Bhopal. “This means no one is being shielded and the probe has been conducted in a fair way.” He declared that the chief minister was completely innocent, and added – very plausibly – that “if senior people had not been arrested, the opposition would have said this is a cover up.”
But what of the deaths? “Please read this booklet,” Shukla said, reaching for a glossy pamphlet titled Vyapam: Propaganda and Reality. “It should answer most of your questions.”
The 23-page booklet, which the state government has distributed widely in Madhya Pradesh and Delhi, praises Chouhan’s administration for its swift and decisive action in appointing a police taskforce to investigate the case, and reviews each fatality alleged to be connected to Vyapam in succinct paragraphs, nearly all of which end with some variation on the following phrase: “The family has not expressed any doubt about his death so far.”
The booklet considers the deaths of 31 people: 11 died in road accidents, five allegedly committed suicide, two drowned in ponds, and three lost their lives to “excessive liquor consumption” – all of which have come under suspicion precisely because of the apparent reluctance of the state police to investigate any of the deaths allegedly connected to the scandal. Shukla and his government, however, insist that all these deaths, while tragic, have no connection to the Vyapam scandal to begin with.
Namrata Damor, the young woman found dead on the railway tracks in 2012, is not mentioned in the government’s list of deaths allegedly linked to Vyapam. When I met her father Mehtab, however, he also insisted that his daughter had no connection to the scam.
But why would anyone want to kill his daughter? “She fell into bad company,” he said, “When a young girl from a small town like Meghnagar goes to a city like Indore, there are always people who could prey on her.”
Yet after our meeting, I spoke to Rahul Karaiya, the local journalist who had gone to interview Damor with the television reporter Akshay Singh shortly before Singh’s death – which remains unsolved. Karaiya sent me a video clip from a “sting operation” that had been conducted by a local TV station four years earlier.
The clip, from 2011 – prior to Damor’s death – records Gaurav Patni, who identifies himself as a fourth-year student at a city medical college, speaking with a reporter on the assumption that his camera was off. Patni claims that he is planning to get out of the business of fixing Vyapam exams because of the increasing difficulty in getting people through and collecting their payments.
“Last year we got only two students through, I’ll even tell you the names,” he says, “One is a girl from Indore, Namrata Damor … You can ask around, they haven’t even paid as yet.”
Was Namrata killed because she did not, or could not, pay whoever had rigged her exam? Her father, understandably, refused to entertain such speculation. He said the police, who had left him waiting years for any news in his daughter’s death, were now floating wild theories to cover up their own incompetence.
The police aren’t the only ones floating theories. The scandal was so vast that almost everyone I met in Madhya Pradesh knew someone connected to it – and it quickly became clear that Vyapam had become the stuff of myth and legend: everyone had a theory, and no scenario was too implausible to entertain.
In an interview with the Hindustan Times earlier this year, a policeman, whose own son was accused in the scam and died in a road accident, advanced an unlikely yet tantalising theory. He argued that the Vyapam taskforce – under pressure to conduct a credible probe that nevertheless absolved top government officials – had falsely named suspects who were already deceased in order to shield the real culprits.
A competing theory, voiced by journalists covering the scandal in Bhopal, proposes that it will be all but impossible to determine whether the deaths are connected to Vyapam, because the families of many of the dead refuse to admit that their children paid money to cheat on their exams – for fear that the police might arrest the bereaved parents as well.
All this suggests that it is unlikely that the truth behind the Vyapam deaths will ever be established. Rather than a simple scam, Vyapam appears to be a vast societal swindle – one that reveals the hollowness at the heart of practically every Indian state institution: inadequate schools, a crushing shortage of meaningful jobs, a corrupt government, a cynical middle class happy to cheat the system to aid their own children, a compromised and inept police force and a judiciary incapable of enforcing its laws.
While the investigation goes nowhere, some of the hundreds of students implicated in the scam have begun to feel like its victims. In Indore, I met a young man who I’ll call Ishan. The son of an impoverished lower-caste family in rural western Madhya Pradesh, neither of his parents can read or write. After moving to Indore in 2007, he spent four years at a series of tuition centres preparing for the medical exam, which he finally passed in 2011. Two years later, when he was a student at medical college, he was swept up in the Vyapam investigation and accused of serving as an agent for one of Jagdish Sagar’s impostors.
“Students preparing for their exams would often approach me for advice,” Ishan told me. “One day a boy asked me for a phone number for a doctor known to Jagdish Sagar. I gave the number because the doctor was our senior from medical school.” Ishan’s friend cleared the exam, was picked up by the police, and mentioned Ishan’s name in the interrogation.
Six years of my life are wasted, my dream of becoming a doctor over. It will take the rest of my life to clear my name
“Now the police claim I am a middleman in the Vyapam scam,” he said, “I spent three months in jail before I was granted bail. My college admission has been cancelled, six years of my life are wasted, and my dream of becoming a doctor is over. I know I will be exonerated, the police have no evidence against me, but it will take the rest of my life to clear my name. Now tell me, why shouldn’t someone in my place commit suicide?”
At a lawyer’s office in Bhopal, I heard a similar tale from a man who had come to help secure bail for his brother, another accused in the scam.
“My brother was arrested four months ago for paying someone to ensure he cleared the police constable exam in 2012,” the man told me. “Some people in our village said, ‘This is Madhya Pradesh, nothing happens without money.’ My brother sold his land and paid them 600,000 rupees.”
In August that year, he was one of 403,253 people who appeared for the recruitment test to become a police constable. When he passed it was clear to everyone that he had a bright future ahead of him – and so he was soon married off. Four months after his marriage, his name popped up in the scam, he lost his job and he was hauled off to prison.
“So now my brother has a wife and his first child, but no job, no land, no money, no prospects and a court case to fight,” the man said. “You can write your story, but write that this is a state of 75 million corrupt people, where there is nothing in the villages and if a man comes to the city in search of an honest day’s work, the politicians and their touts demand money and then throw him into jail for paying.”
It seemed plausible that some of Vyapam deaths really were suicides – that people did hang themselves, jump off trains, and drink themselves to death rather than meeting their demise at the hands of mysterious assassins. Many of the accused had, at great personal expense, through fraud or perseverance, succeeded in overcoming a system designed to reward a microscopic minority with the lifelong privilege of permanent employment, only to see their rewards snatched away after the fact. But these deaths were not “unrelated to Vyapam”, as the government kept insisting. Rather, they seemed a consequence of the prevailing corruption common to both the scandal and and investigation that shows no sign of ever concluding.
“A scam like this is going to take years to investigate,” a lawyer representing several of the thousands of accused told me. “The CBI just doesn’t have the manpower to investigate so many deaths and arrests. When the CBI took the case from the police, they literally sent a truck to gather all the documents.”
So the case, in all likelihood, will ultimately collapse into a giant, disorganised pile of court hearings and paperwork. Memories of the dead will fade away, while the living spend the rest of their lives appearing in court to defend themselves from the accusation that they once cheated on their exams.
On 9 October this year, the Supreme Court reviewed the progress of the Vyapam case with some satisfaction, noting that the mysterious deaths had suddenly ceased since the investigation was taken away from the Madhya Pradesh police and handed to the CBI.
“I was wondering that from the time the court had ordered the CBI probe into the case and decided to monitor the investigations,” the chief justice, HL Dattu, asked, “how come not a single death has been reported?”
One week later, the driver of a train en route to Bhopal spotted a corpse on the tracks. The body was later identified as Vijay Bahadur, a retired Madhya Pradesh bureaucrat, who had served as an observer for at least two Vyapam exams. His wife, who had been travelling with him, told the police that Bahadur had stepped out of their train compartment and into the corridor to shut the door of their carriage, and never returned. The CBI has now added Bahadur to the list of suspected Vyapam-related fatalities – and begun an investigation into his death.
• Follow the Long Read on Twitter at @gdnlongread, or sign up to the long read weekly email here.
POSTED BY BOB WORRALL, SVP CHIEF INFORMATION OFFICER ON DECEMBER 17, 2015
Juniper is committed to maintaining the integrity and security of our products and wanted to make customers aware of critical patched releases we are issuing today to address vulnerabilities in devices running ScreenOS® software.
During a recent internal code review, Juniper discovered unauthorized code in ScreenOS that could allow a knowledgeable attacker to gain administrative access to NetScreen® devices and to decrypt VPN connections. Once we identified these vulnerabilities, we launched an investigation into the matter, and worked to develop and issue patched releases for the latest versions of ScreenOS.
At this time, we have not received any reports of these vulnerabilities being exploited; however, we strongly recommend that customers update their systems and apply the patched releases with the highest priority.
On behalf of the entire Juniper Security Response Team, please know that we take this matter very seriously and are making every effort to address these issues. More information and guidance on applying this update to systems can be found in the Juniper Security Advisories (JSAs) available on our Security Incident Response website at http://advisory.juniper.net.
Bob Worrall
SVP Chief Information Officer
Q: Why did this issue require an out-of-cycle security advisory?
Juniper is committed to maintaining the integrity and security of our products. Consistent with industry best practices, this means releasing patches for products in a timely manner to maintain customer security. We believed that it was in our customers’ best interest to issue these patched releases with the highest priority.
We strongly recommend that all customers update their systems and apply these patched releases as soon as possible.
Q: What devices does this issue impact?
All NetScreen devices using ScreenOS 6.2.0r15 through 6.2.0r18 and 6.3.0r12 through 6.3.0r20 are affected by these issues and require patching. We strongly recommend that all customers update their systems and apply these patched releases with the highest priority.
Q: Is the SRX or any other Junos®-based system affected by these issues?
These vulnerabilities are specific to ScreenOS. We have no evidence that the SRX or other devices running Junos are impacted at this time.
Q: Who can I contact if I have additional questions about my system?
Customers with questions about their systems should e-mail us at sirt@juniper.net
This domain name is pending ICANN verification and has been suspended. If you are the owner of this domain you can reactivate this domain by logging into your EuroDNS account.
REACTIVATE DOMAIN
As of 1st January 2014, ICANN, the governing body for domain names, requires that the contact details provided to register a domain are validated at least once a year.
This domain has been suspended for one of the following reasons:Posted on Dec. 18, 2015
The PostgreSQL Global Development Group announces that the first release candidate is available for download. This release should fix all outstanding issues with PostgreSQL 9.5, and be identical to the final release, pending bug fixes. All users intending to upgrade to PostgreSQL 9.5 should download and try PostgreSQL 9.5 RC1 now.
All outstanding issues and known bugs in version 9.5 have been fixed in this release. As such, any issues you encounter are new and should be reported to us. Fixes since Beta2 include:
If you reported an issue while testing PostgreSQL 9.5, please download RC1 and test whether that issue has been fixed. If you have not yet tested version 9.5, now is the time for you to help out PostgreSQL development.
This is the first release candidate of 9.5, indicating that it is expected to be identical to the final release. If new major issues are found, the PostgreSQL project will release additional release candidates. Otherwise, 9.5.0 will be released in early January. For more information, and suggestions on how to test the release candidate, see the Beta Testing page.
Full documentation and release notes of the new version is available online and also installs with PostgreSQL. Also see the What's New page for details on some features.
Previously I wrote about clarity being the most important characteristic of a great interface. Let’s talk about icons now. They’re an essential part of many user interfaces. The thing is: more often than not, they break clarity.
Pictograms have been in use since the early days of mankind. They are often seen as the first expressions of a written language. Some non-literate cultures still use them today as their main medium of written communication.
In many public spaces, they’re used for wayfinding. Especially in a multi-cultural environment like an airport, traditional written language would not suffice.
Icons soon became popular in user interface design too. Have a look at this shot of the first commercial graphical user interface on a desktop computer (the Xerox Star). Designer David Smith invented the concept of icons and the desktop metaphor:
Of course I can see why icons grew popular in user interfaces. Firstly, they make the UI more graphically pleasing. And when done right, they can certainly give your app visual personality. That’s two good things.
Moreover, an icon can often replace a long descriptive group of words. As screens get smaller, this is much welcomed. But herein lies the design trap, because most icons are unclear. They make people think. What good has a beautiful interface if it’s unclear? Hence it’s simple: only use an icon if its message is a 100% clear to everyone. Never give in.
(Unclear icons remind me of this funny graphic someone posted on Twitter recently:
I guess Ron got frustrated by all these unreadable washing icons…)
Discussing the use of icons with clients, the remark I hear often goes like this: “People will use our software daily, they’ll soon learn the meaning of our icons.” Makes sense, but I’m not sure that’s entirely correct. I use Apple mail several times a day, yet still feel uncertain which icon to click when composing a new message:
And I have more bad news: users will avoid interface elements they cannot understand. It’s in our nature to distrust the unknown. When Google decided to hide other apps behind an unclear icon in the Gmail UI, they apparently got a stream of support requests, like “Where is my Google Calendar?”
Another example: after a redesign, new Twitter users plausibly didn’t understand what they were supposed to do. The icon simply wasn’t clear enough:
A later version made tweeting much more obvious:
Did you know you can send a direct private photo message to an Instagram friend? It’s behind this pictogram:
Hmmm… as an Instagram user, chances are you’ve seen it before, but have you ever used it? It’s unclear. What does it mean?
Of course context is important too when using icons. Some crystal clear pictograms get ambiguous in a different context. Watch out for that. Upon opening a Gmail email conversation, you get to see this. See how those two icons are very similar, but have a different meaning?
I’ve sure arrived in my inbox more than once when all I wanted to do was reply to an email. Context matters.
Tweetbot’s icons may look unclear to some of us, they are clear in the context of Twitter. Tweetbot users are Twitter users, they will recognize these so it seems okay to me to use them here:
The same can be said about the Tumblr iOS app. Some of these icons may look weird to most of us, they’re pretty clear in the context of Tumblr. Its users will recognize these.
The Rdio Mac app looks like this:
Most of the icons in use here are clear in the context of a music player. (A weird thing though is that one icon has 2 meanings: the volume and the currently played song.)
So let me repeat: don’t use an icon if its meaning isn’t a 100% clear to everyone. When in doubt, skip the icon. Reside to simple copy. A text label is always clearer.
If you want to keep the graphical advantages of icons, you can of course combine the icon with copy. It’s an excellent solution that unites the best of both worlds.
The Mac App Store is doing exactly this. It’s almost mandatory here, because the icons themselves would be totally unclear:
It’s also what Twitter is doing in their web interface:
Let’s look at Facebook as a final example: they lately traded their unclear hamburger menu icon for a frictionless navigation that combines icons with clear copy. Well done:
I hope all of this made clear that icons can easily break the most important characteristic of a good user interface: clarity. So be very careful, and test! And when in doubt, always remember this: the best icon is a text label.
Thomas Byttebier (@bytte) — March 23, 2015
Open Hunt is a brand new community for fans and builders of early stage technology products. We aim to be completely open and transparent, without insiders or gatekeepers who arbitrarily control who gets listed and who gets left out.
Our initial design may look similar to another site you know of. This is intentional for the first launch, but not at all final. There are already some key differences, and we'll evolve even more as we grow. You can get involved: file an issue or submit a pull request. We would love your help!
This community is just starting and still very fragile. Please bear with us for the initial bumps and bruises. Our content may be sparse to begin with, but with your support, we can create an alternative product enthusiast community that lasts.
Thank you for your support!
-- The Open Hunt team
Apparently I’ve earned the dubious distinction of having become an expert in project failure. I’ve always had an interest in project failure—Jenny and I have been doing our “Why Projects Fail” talk for years now, and we’ve talked to many people in many different industries (like in our fourth book, Beautiful Teams) about what’s gone wrong on their projects. We’ve looked at failures on projects through the years, from small misfires on our own projects to dramatic failures like the Tacoma Narrows Bridge disaster, to try to figure out what software developers can learn from them.
One of my favorite ways that projects can fail is death by micromanagement. It’s a nasty, insidious problem for a couple of reasons. It’s easy for a boss to fall into the micromanagement trap, especially for a project that’s starting to spiral out of control, because when you feel like your project is slipping away from you, it’s hard to resist the urge to tighten your grip on every part of it that you can.
And the people on the team have trouble recognizing it because a lot of them have never worked any other way. I’ve said it before, and I’m sure I’ll say it again: I’m willing to bet that if someone was able to conduct an accurate survey, they would find that a surprisingly large number of managers are micromanagers.
On the other hand, if you’re a boss or a project manager looking for a great way to demoralize your team and cause your projects to fail, micromanagement is a great way to do it. Here are some handy tips to make sure your team hates you and your project runs into serious trouble:
Most of all, though, remember rule #1: Nobody is ever allowed to make mistakes! If a developer makes a mistake, it reflects badly on you, and the whole team suffers. Never admit that you were wrong about anything. If you said it once, it’s now the law and should never be questioned.
If you follow this simple advice, then your team will be demoralized in no time. Also, they’ll hate you. Oddly, though, there’s a good chance that they won’t get their resumes together and start looking for new jobs. I have a theory about this: when you micromanage a team, it drains their will to such an extent that they no longer care. Psychologists call this state “learned helplessness.” The Wikipedia article on learned helplessness has a good description of a classic experiment by Martin Seligman and Steve Maier:
In part one of Seligman and Steve Maier‘s experiment, three groups of dogs were placed in harnesses. Group One dogs were simply put in the harnesses for a period of time and later released. Groups Two and Three consisted of “yoked pairs.” A dog in Group 2 would be intentionally subjected to pain by being given electric shocks, which the dog could end by pressing a lever. A Group 3 dog was wired in parallel with a Group 2 dog, receiving shocks of identical intensity and duration, but his lever didn’t stop the electric shocks. To a dog in Group 3, it seemed that the shock ended at random, because it was his paired dog in Group 2 that was causing it to stop. For Group 3 dogs, the shock was apparently “inescapable.” Group 1 and Group 2 dogs quickly recovered from the experience, but Group 3 dogs learned to be helpless, and exhibited symptoms similar to chronic clinical depression.
In part two of the Seligman and Maier experiment, these three groups of dogs were tested in a shuttle-box apparatus, in which the dogs could escape electric shocks by jumping over a low partition. For the most part, the Group 3 dogs, who had previously “learned” that nothing they did had any effect on the shocks, simply lay down passively and whined. Even though they could have easily escaped the shocks, the dogs didn’t try.
If reading about the Group 3 dogs reminds you of work, then there’s a very good chance that you’re being micromanaged.
Encryption backdoors have been a hot topic in the last few years—and the controversial issue got even hotter after the terrorist attacks in Paris and San Bernardino, when it dominated media headlines. It even came up during this week’s Republican presidential candidate debate. But despite all the attention focused on backdoors lately, no one noticed that someone had quietly installed backdoors three years ago in a core piece of networking equipment used to protect corporate and government systems around the world.
On Thursday, tech giant Juniper Networks revealed in a startling announcement that it had found “unauthorized” code embedded in an operating system running on some of its firewalls.
The code, which appears to have been in multiple versions of the company’s ScreenOS software going back to at least August 2012, would have allowed attackers to take complete control of Juniper NetScreen firewalls running the affected software. It also would allow attackers, if they had ample resources and skills, to separately decrypt encrypted traffic running through the Virtual Private Network, or VPN, on the firewalls.
“During a recent internal code review, Juniper discovered unauthorized code in ScreenOS that could allow a knowledgeable attacker to gain administrative access to NetScreen devices and to decrypt VPN connections,” Bob Worrall, the companies’ CIO wrote in a post. “Once we identified these vulnerabilities, we launched an investigation into the matter, and worked to develop and issue patched releases for the latest versions of ScreenOS.”
'This is a very good showcase for why backdoors are really something governments should not have in these types of devices because at some point it will backfire.'
Juniper released patches for the software yesterday and advised customers to install them immediately, noting that firewalls using ScreenOS 6.2.0r15 through 6.2.0r18 and 6.3.0r12 through 6.3.0r20 are vulnerable. Release notes for 6.2.0r15 show that version being released in September 2012, while release notes for 6.3.0r12 show that the latter version was issued in August 2012.
The security community is particularly alarmed because at least one of the backdoors appears to be the work of a sophisticated nation-state attacker.
“The weakness in the VPN itself that enables passive decryption is only of benefit to a national surveillance agency like the British, the US, the Chinese, or the Israelis,” says Nicholas Weaver, a researcher at the International Computer Science Institute and UC Berkeley. “You need to have wiretaps on the internet for that to be a valuable change to make [in the software].”
But the backdoors are also a concern because one of them—a hardcoded master password left behind in Juniper’s software by the attackers—will now allow anyone else to take command of Juniper firewalls that administrators have not yet patched, once the attackers have figured out the password by examining Juniper’s code.
Ronald Prins, founder and CTO of FOX-IT, a Dutch security firm, said the patch released by Juniper provides hints about where the master password backdoor is located in the software. By reverse-engineering the firmware on a Juniper firewall, analysts at his company found the password in just six hours.
“Once you know there is a backdoor there, … the patch [Juniper released] gives away where to look for [the backdoor] … which you can use to log into every [Juniper] device using the Screen OS software,” he told WIRED. “We are now capable of logging into all vulnerable firewalls in the same way as the actors [who installed the backdoor].”
But there is another concern raised by Juniper’s announcement and patches—any other nation-state attackers, in addition to the culprits who installed the backdoors, who have intercepted and stored encrypted VPN traffic running through Juniper’s firewalls in the past, may now be able to decrypt it, Prins says, by analyzing Juniper’s patches and figuring out how the initial attackers were using the backdoor to decrypt it.
“If other state actors are intercepting VPN traffic from those VPN devices, … they will be able to go back in history and be able to decrypt this kind of traffic,” he says.
Weaver says this depends on the exact nature of the VPN backdoor. “If it was something like the Dual EC, the backdoor doesn’t actually get you in, … you also need to know the secret. But if it’s something like creating a weak key, then anybody who has captured all traffic can decrypt.” Dual EC is a reference to an encryption algorithm that the NSA is believed to have backdoored in the past.
Matt Blaze, a cryptographic researcher and professor at the University of Pennsylvania, agrees.
“If the VPN backdoor doesn’t require you to use the other remote-access [password] backdoor first,” then it would be possible to decrypt historical traffic that had been captured, he says. “But I can imagine designing a backdoor in which I have to log into the box using the remote-access backdoor in order to enable the backdoor that lets me decrypt intercepted traffic.”
A page on Juniper’s web site does appear to show that it’s using the weak Dual EC algorithm in some products, though Matthew Green, a cryptography professor at Johns Hopkins University, says it’s still unclear if this is the source of the VPN issue in Juniper’s firewalls.
Juniper released two announcements about the problem on Thursday. In a second more technical advisory, the company described two sets of unauthorized code in the software, which created two backdoors that worked independently of one another, suggesting the password backdoor and the VPN backdoor aren’t connected. A Juniper spokeswoman refused to answer questions beyond what was already said in the released statements.
Regardless of the precise nature of the VPN backdoor, the issues raised by this latest incident highlight precisely why security experts and companies like Apple and Google have been arguing against installing encryption backdoors in devices and software to give the US government access to protected communication.
“This is a very good showcase for why backdoors are really something governments should not have in these types of devices because at some point it will backfire,” Prins says.
Green says the hypothetical threat around NSA backdoors has always been: What if someone repurposed them against us? If Juniper did use Dual EC, an algorithm long-known to be vulnerable, and this is part of the backdoor in question, it underscores that threat of repurposing by other actors even more.
“The use of Dual EC in ScreenOS … should make us at least consider the possibility that this may have happened,” he told WIRED.
The first backdoor Juniper found would give an attacker administrative-level or root privileges over the firewalls—essentially the highest-level of access on a system—when accessing the firewalls remotely via SSH or telnet channels. “Exploitation of this vulnerability can lead to complete compromise of the affected system,” Juniper noted.
Although the firewall’s log files would show a suspicious entry for someone gaining access over SSH or Telnet, the log would only provide a cryptic message that it was the “system” that had logged on successfully with a password. And Juniper noted that a skilled attacker would likely remove even this cryptic entry from log files to further eliminate any indication that the device had been compromised.
The second backdoor would effectively allow an attacker who has already intercepted VPN traffic passing through the Juniper firewalls to decrypt the traffic without knowing the decryption keys. Juniper said that it had no evidence that this vulnerability had been exploited, but also noted that, “There is no way to detect that this vulnerability was exploited.”
Juniper is the second largest maker of networking equipment after Cisco. The Juniper firewalls in question have two functions. The first is to ensure that the right connections have access to a company or government agency’s network; the other is to provide secured VPN access to remote workers or others with authorized access to the network. The ScreenOS software running on Juniper firewalls was initially designed by NetScreen, a company that Juniper acquired in 2004. But the versions affected by the backdoors were released under Juniper’s watch, eight years after that acquisition.
The company said it discovered the backdoors during an internal code review, but it didn’t say if this was a routine review or if it had examined the code specifically after receiving a tip that something suspicious was in it.
Speculation in the security community about who might have installed the unauthorized code centers on the NSA, though it could have been another nation-state actor with similar capabilities, such as the UK, China, Russia, or even Israel.
Prins thinks both backdoors were installed by the same actor, but also notes that the hardcoded master password giving the attackers remote access to the firewalls was too easy to find once they knew it was there. He expects the NSA would not have been so sloppy.
Weaver says it’s possible there were two culprits. “It could very well be that the crypto backdoor was [done by] the NSA but the remote-access backdoor was the Chinese or the French or the Israelis or anybody,” he told WIRED.
NSA documents released to media in the past show that the agency has put a lot of effort into compromising Juniper firewalls and those made by other companies.
An NSA spy tool catalogue leaked to Der Spiegel in 2013 described a sophisticated NSA implant known as FEEDTHROUGH that was designed to maintain a persistent backdoor in Juniper firewalls. FEEDTHROUGH, Der Spiegel wrote, “burrows into Juniper firewalls and makes it possible to smuggle other NSA programs into mainframe computers…..” It’s also designed to remain on systems even after they’re rebooted or the operating system on them is upgraded. According to the NSA documents, FEEDTHROUGH had “been deployed on many target platforms.”
FEEDTHROUGH, however, appears to be something different than the unauthorized code Juniper describes in its advisories. FEEDTHROUGH is a firmware implant—a kind of “aftermarket” spy tool installed on specific targeted devices in the field or before they’re delivered to customers. The unauthorized code Juniper found in its software was embedded in the operating system itself and would have infected every customer who purchased products containing the compromised versions of the software.
Naturally, some in the community have questioned whether these were backdoors that Juniper had voluntarily installed for a specific government and decided to disclose only after it became apparent that the backdoor had been discovered by others. But Juniper was quick to dispel those allegations. “Juniper Networks takes allegations of this nature very seriously,” the company said in a statement. “To be clear, we do not work with governments or anyone else to purposely introduce weaknesses or vulnerabilities into our products… Once this code was discovered we worked to produce a fix and notify customers of the issues.”
Prins says the larger concern now is whether other firewall manufacturers have been compromised in a similar manner. “I hope that other vendors like Cisco and Checkpoint are also now starting a process to review their code to see if they have backdoors inserted,” he said.
Go Back to Top. Skip To: Start of Article.AI: computers that ‘think’ spell doom for many lawyers
Robots and artificial intelligence (AI) will dominate legal practice within 15 years, perhaps leading to the “structural collapse” of law firms, a report predicting the shape of the legal market has envisaged.
Civilisation 2030: The near future for law firms, by Jomati Consultants, foresees a world in which population growth is actually slowing, with “peak humanity” occurring as early as 2055, and ageing populations bringing a growth in demand for legal work on issues affecting older people.
This could mean more advice needed by healthcare and specialist construction companies on the building and financing of hospitals, and on pension investment businesses, as well as financial and regulatory work around the demographic changes to come; more age-related litigation, IP battles between pharmaceutical companies, and around so-called “geriatric-tech” related IP.
The report’s focus on the future of work contained the most disturbing findings for lawyers. Its main proposition is that AI is already close in 2014. “It is no longer unrealistic to consider that workplace robots and their AI processing systems could reach the point of general production by 2030… after long incubation and experimentation, technology can suddenly race ahead at astonishing speed.”
By this time, ‘bots’ could be doing “low-level knowledge economy work” and soon much more. “Eventually each bot would be able to do the work of a dozen low-level associates. They would not get tired. They would not seek advancement. They would not ask for pay rises. Process legal work would rapidly descend in cost.”
The human part of lawyering would shrink. “To sustain margins a law firm would have to show added value elsewhere, such as in high-level advisory work, effectively using the AI as a production tool that enabled them to retain the loyalty and major work of clients…
“Clients would instead greatly value the human input of the firm’s top partners, especially those that could empathise with the client’s needs and show real understanding and human insight into their problems.”
Jomati pointed out that the managing partners of 2030 are in their 30s today and will embrace the advantages of AI. Alternative business structures (ABSs) in particular will be receptive, it predicted, “as it will greatly suit the type of matters they handle”.
It continued: “With their external investors able to provide significant capital, they will invest in the latest AI when it becomes available and use it to rapidly increase the volume of matters. This increased efficiency will not harm their model, but rather make the shareholders in their narrow equity model extremely wealthy.”
For associate lawyers, the rise of AI will be a disaster: “The number of associates that firms need to hire will be greatly reduced, at least if the intention is to use junior lawyers for billable work rather than primarily to educate and train them ready to become business winners.
“Firms will struggle to overcome this gap in the usual career paths of their lawyers, i.e. firms need to hire young lawyers to become the next client winners, but they will be far less profitable at the start of their careers when knowledge bots take over most work up to [three years’] PQE.”
On the impact of AI on law firms, Jomati concluded: “The economic model of law firms is heading for a structural revolution, some might say a structural collapse. We may have heard a lot about ‘New Law’ and [ABS], but the impact of AI will make such developments pale in comparison.”
Small, specialist advisory firms and those focused on process matters might not be affected by AI, the report predicted, adding: “The firms that will be most affected would be the very large, high-value commercial firms whose associates expect to be given interesting work and many of whom aspire to the status and wealth that equity partnership affords. These fee-earners are also incredibly profitable as they clock up the hours on matters the partners have brought in.”
The report forecast a big rise in the number of cities of over 10m people. Jomati’s top five cities in 2030 for their legal markets were New York, London, Paris, Frankfurt, and Singapore. It anticipated: “Far greater global balance in the largest law firms as they seek to follow clients into developing markets and key megacities and global cities around the world. While some global firms already have more than 50% of their revenues and staff based outside the ‘home nation’, by 2030 this will become standard for nearly all major commercial firms.”
Tags: artificial intelligence, legal technology
During my journey into microservices, it has become apparent that the majority of online sample/howto posts regarding implementation focus solely on REST as a means for microservices to communicate with each other. Because of this, you could be forgiven for thinking that RESTful microservices are the de facto standard, and the approach to strive for when designing and implementing a microservices-based system. This is not necessarily the case.
The reason why REST based microservices examples are most popular is more than likely due to their simplicity; services communicate directly and synchronously with each other over HTTP, without the need for any additional infrastructure.
As an example consider a system that notifies customers when a particular item is back in stock. This could be implemented via RESTful microservices as so:
It should be noted that although communication is point-to-point, hard coding services addresses would be a very bad design choice, and goes against the very fundamentals of microservices. Instead a Service Discovery mechanism should be used, such as Eureka or Consul, where services register their API availability with a central server, and clients to request a specific API address from this server.
Diving deeper, there are a number of fundamental flaws or things to consider in this implementation:
Due to the synchronous nature of REST, the update stock operation will not return until the notification service has completed its task of notifying all relevant customers. Imagine the effect of this if a particular item is very popular, with 1000s of customers wishing to be notified of the additional stock. Performance could potentially be severely affected and the scalability of the system will be hindered.
The knowledge that ‘when an item is back in stock, customers should be notified’ is ingrained into the inventory manager service but I would argue that this should not be the case. The single responsibility of this service should be to update system inventory (the inventory aggregate root) and nothing else. In fact, it should not even need to know about the existence of the notification service at all. The two services are tightly coupled in this model.
Services WILL fail, and a microservice based system should continue to function as well as possible during these situations. Due to the tightly coupled system described above, there needs to be a failure strategy within the Inventory Manager (for example) to deal with the scenario where the Back in Stock Notifier is not available. Should the inventory update fail? Should the service retry? It is also vital that the request to the Notifier fails as quickly as possible, something that the circuit breaker pattern (e.g. Hystrix) can help with. Even though failure scenarios will have to be handled regardless of the communication method, bundling all this logic into the calling service will add bloat. Coming back to the single-responsibility issue, again it’s my opinion that the Inventory Manager should not be responsible for dealing with the case where the Notifier goes dark.
One way to overcome the coupling of services and to move the responsibility of routing away from a microservice is to follow the pipeline enterprise pattern. Our subsystem would now look like this:
Communication may still be REST based, but is no longer ‘point-to-point’; it is now the responsibility of the pipeline entity to orchestrate the data flows, rather than the services themselves. Whilst this overcomes the coupling issue (and blocking with a bit more work, via asynchronous pipelines), it is considered good practice within the microservices community to strive for services that are as autonomous and coherent as possible. With this approach, the services must rely on a third party entity (the pipeline orchestrator) in order to function as a system and are therefore not particularly self sufficient.
For example, notice that the pipeline will receive a single response from the Back in Stock Notifier (even though there are 2 subscribers), but must be configured in such a way that it can parse the response so that it can subsequently send individual “send email” requests to the Email Notifier for each subscriber. It could be argued that the Email Sender could be modified to batch send emails to many different subscribers via a single request, but if for example, each users name must be included in the email body, then there would have to be some kind of token replace functionality. This introduces additional behavioural coupling, where the Notifier has specific knowledge about the behaviour of the Email Sender.
In a messaging based system, both the input and output from services are defined as either commands or events. Each service subscribes to the events that it is interested in consuming, and then receives these events reliably via a mechanism such as a messaging queue/broker, when the events are placed on the queue by other services.
Following this approach, the stock notification subsystem could now be remodelled as follows:
Cohesion is obtained via a shared knowledge of queue names, and a consistent and well known command/event format; an event or command fired by one service should be able to be consumed by the subscriber services. In this architecture, a great deal of flexibility, service isolation and autonomy is achieved.
The Inventory Manager for instance, has a single responsibility, updating the inventory, and is not concerned with any other services that are triggered once it has performed its task. Therefore, additional services can be added that consume Inventory Updated events without having to modify the Inventory Manager Service, or any pipeline orchestrator.
Also, it really doesn’t care (or have any knowledge of) if the Back in Stock Notifier has died a horrible death; the inventory has been updated so it’s a job well done, as far as the Inventory Manager is concerned. This obliviousness of failure from the Inventory Manager service is actually a good thing; we MUST still have a strategy for dealing with the Back in Stock Notifier failure scenario, but as I have stated previously, it could be argued that this is not the responsibility of the Inventory Manager itself.
This ability to deal with change such as adding, removing or modifying services without affecting the operation or code of other services, along with gracefully handling stressors such as service failure, are two of the most important things to consider when designing a microservices based system.
Everything in the world of asynchronous messaging isn’t entirely rosy however, and there are still a few pitfalls to consider:
The programming model is generally more complex in an asynchronous system compared to a synchronous counterpart, making it more difficult to design and implement. This is because there are a number of additional issues that may have to be overcome, such as message ordering, repeat messages and message idempotency.
Also, the configuration of the message broker will also need some thought. For example, if there are multiple instances of the same service, should a message be delivered to both of the services or just one? There are use cases for both scenarios.
The fact that the result of an action is not returned immediately can also increase the complexity of system and user interface design and in some scenarios it does not even make logical sense for a subset of a system to function in an asynchronous manner. Take the Back in Stock Notifier for example, and its relationship with the Subscriber Manager; it is impossible for the notifier to function without information about the subscribers that it should be notifying, and therefore a synchronous REST call makes sense in this case. This differs from the email sending task, as there is no need for emails to be sent immediately.
Due to the dispersed and autonomous nature of messaging based microservices, it can be difficult to fully get a clear view of the flow of messages within the system. This can make debugging more difficult, and compared to the pipeline approach, the business logic of the system is harder to manage.
Note: Event based messaging can expanded even further by applying event-sourcing and CQRS patterns, but this is beyond the scope of this article. See the further reading links for more information.
So which communication approach is best when designing your microservices? As with most things in software development (and life??), it depends on the requirements! If a microservice has a real need to respond synchronously, or if it needs to receive a response synchronously itself, then REST may well be the approach that you would want to take. If an enterprise requires the message flow through the system to be easily monitored and audited, or if it’s considered beneficial to be able to modify and view the flow through the system from one centralised place, then consider a pipeline. However, the loosely coupled, highly scalable nature of asynchronous messaging based systems fits well with the overall ethos of microservices. More often than not, despite some significant design and implementation hurdles, an event based messaging approach would be a good choice when deciding upon a default communication mechanism in a microservices based system.
The Internet Archive is hosting a live, 24-hour Telethon during our end-of-year fundraising period! With a variety of guests, activities and surprises, we hope you'll make a donation to our goal and watch the fun!
Your hosts are Jason and Michelle, and they are going to be providing non-stop entertainment and education about the mission and goal of the Internet Archive from Noon on December 19th through to Noon on December 20th.
We're doing this to bring attention to the treasure of the Internet Archive, which for nearly 20 years has been involved in history (through our Wayback Machine), collections of items (such as our books, music, movies, and software), and analysis of television news. Millions visit the Archive's pages every year, for research, entertainment, education, and insight. We're hoping you'll be inspired to make this year's fundraising our best yet!
The Internet Archive is a 501(c)(3) non-profit that was founded to build an Internet library. Its purposes include offering permanent access for researchers, historians, scholars, people with disabilities, and the general public to historical collections that exist in digital format.
Founded in 1996 and located in San Francisco, the Archive has been receiving data donations from Alexa Internet and others. In late 1999, the organization started to grow to include more well-rounded collections. Now the Internet Archive includes: texts, audio, moving images, and software as well as archived web pages in our collections, and provides specialized services for adaptive and accessible reading.
The Internet Archive is working to prevent the Internet - a new medium with major historical significance - and other "born-digital" materials from disappearing into the past. Collaborating with institutions including the Library of Congress and the Smithsonian, we are working to preserve a record for generations to come. Open and free access to literature and other writings has long been considered essential to education and to the maintenance of an open society. Public and philanthropic enterprises have supported it through the ages. The Internet Archive is opening its collections to researchers, historians, and scholars.
Former AIG chairman Hank Greenberg in Washington, DC, prior to testifying before the House Oversight Committee in April 2009. Photo by Gerald Herbert/AP
In case you were born in the 90s and don't remember the infamous AIG bonus controversy of March 2009, it was a bizarre moment in our nation's history when it seemed like everyone in America was angry about the same thing. And that particular thing was the crux of the "problem"—a microcosm of everything else that had gone wrong with Western civilization. Today, thanks to a lawsuit being argued in Washington, we now know how comprehensively we were duped.
AIG is an insurance company that sold credit-default swaps—insurance policies on the default risk of companies, countries, and other issuers of bonds. Some of those bonds were amalgamations of agglomerations of very dubious mortgages, which is in part why AIG ended up famously needing a massive federal bailout in autumn 2008. A few months later, someone in Washington noticed a provision in the bailout legislation that ensured retention bonuses for certain AIG employees, $165 million of which were being paid out to employees of the financial products division. Within days, various legislators had subpoenaed the company for the names and addresses of the bonus recipients. They introduced legislation levying a 90 percent tax on any bonuses that were not returned to the government; convened numerous congressional hearings in the ostensible attempt to hold someone accountable for this treasonous waste of taxpayer dollars; and deluged the 24-hour news cycle with inflammatory sound bites advising bonus recipients to resign, return the money, or kill themselves.
Reading news coverage of all this five and a half years later is kind of like reading old infatuation-era emails with a much older ex-boyfriend you never think about—but who in hindsight was kind of a pedophile. It goes without saying you will never again be that young or trusting, your senses never again so alive as they were in those months after the election of 2008, when it seemed as though our elected officials had somehow finally been moved by unprecedented events to switch their respective moral compasses off airplane mode and tax the rich. But it's hard not to feel a little gross for having personally joined the outrage orgy—especially given the revelations of the lawsuit against the government brought by former AIG CEO Hank Greenberg, a cranky old rich guy Tim Geithner definitely figured would be dead by now.
At the time we were told that it was the "greatest corporate failure in American history"—hat tip to Senator Richard Shelby—because politicians needed to cover up for the actual perpetrators of the crisis. They had us believe that AIG's "rogue" financial products unit was the assassin of the economy, because they were conveniently placed and relatively unsympathetic bit players who happened to be completely removed from the more sinister machinations of the subprime racketeers. And in stark contrast to the narratives promulgated by former Treasury secretaries Tim Geithner and Henry Paulson and by appointed media mouthpieces like Andrew Ross Sorkin—whose crisis chronicle Too Big To Fail portrays the AIG bailout as the thoughtless design of a group of bankers Geithner summoned to the New York Federal Reserve Bank and exhorted to "Work harder, get smarter!"—the truth is that it was all very deliberate and intentional. Because, as the Fed and Treasury lawyers were acknowledging among themselves at the time, the AIG "bailout" appears to be by far the most illegal thing they did.
Here's how Greenberg's complaint chronicles his company's assassination: On September 16, 2008, Geithner called Bob Willumstad, who had been the CEO of the company for all of three months, and told him he was about to send him the term sheet for an emergency line of credit, that he wasn't going to like the terms, and that that was OK, because they were firing him anyway—replacing him with someone they could persuade to take the job. (Naturally, that someone, Ed Liddy, would turn out to be a former Goldman Sachs board member.) But the Fed apparently hadn't gotten around to drafting a term sheet, so instead it furnished AIG's outside counsel, a ubiquitous crisis figure named H. Rodgin Cohen, with two pages of incoherent bullet points Willumstad later said "might have been put together by my grandchildren." Cohen instructed Willumstad to sign a single page with a line for his signature and the date, and nothing attached. He faxed it over to the Fed.
For the next week the AIG board labored under the delusion that they were negotiating a deal with the government—even filing a notice with the SEC on September 18 announcing their intention to hold a shareholder meeting to vote on the terms of the deal "as soon as practicable."
Meanwhile, Fed lawyers were exchanging emails about how they were going to legally cover their asses to avoid any such vote, since there was decisively no provision in the Fed charter allowing it to seize control of private institutions. Finally, Thomas Baxter, the general counsel of the New York Fed, decided it would be OK if some of them set up an independent trust, overruling the objections of his titular superiors because, he explained in an email on September 21, the "concern that there will be a shareholder action."
Cohen presented the government's terms to the board that evening. It would extend to AIG an $85 billion line of credit at an interest rate of 14.5 percent, in exchange for $85 billion in collateral and 79.9 percent of the company's shares. He would also no longer countenance AIG filing for bankruptcy under the "business judgment rule"—leaving the directors open to litigation if they voted to file. Which is when they realized, as one director put it, that "the government stole at gunpoint 80 percent of the company."
Once in control, the government went about engineering AIG's blitzkrieg self-destruction. The 14.5 percent interest rate was a death sentence in itself—the other 407 institutions that borrowed from the Fed during the crisis paid an average of less than 2 percent. But then the Fed commanded its puppet CEO to immediately buy back some $62 billion of mystery-meat mortgage "products" from a consortium of megabanks (but especially Goldman, which controlled $22 billion in mortgage credit-default swaps) at 100 percent of their face value. To put this in perspective, banks that purchased credit protection from insurers other than AIG ended up getting as little as 13 cents on the dollar out of the ensuring bankruptcies. If AIG had been allowed to file, it might have fared even worse than that, because in many cases the mystery meat it received in this deal was far more diseased than the crap it had actually insured.
In spite of all this, AIG ultimately returned to profitability. In fact, the relative health of its business may have fucked it harder than all those credit-default swaps—as one of the last institutions to run out of cash, it was also one of the last to start seeking a lifeline from foreign wealth funds. Once it did, the China Investment Corporation, the Government of Singapore Investment Corporation, and a consortium of Arabs who got Hillary Clinton to lobby their case all approached Paulson with serious offers. All were rebuffed. On September 26, China Inc. made an explicit $50 billion offer for four subsidiaries. It took ten days for Paulson to even bother calling back to officially blow them off.
Given all the blatantly illegal actions the government took to commandeer control of AIG, the bonus controversy looks almost like a cheap ploy to give the appearance that the Treasury and Fed felt somehow bound by the law in their dealings with the company. At the same time, as AIG executives' houses became destinations for populist rage tourism, the uproar served to further intimidate anyone at the company who might have questioned the government's handling of the situation.
Former US Treasury Secretary Timothy Geithner testified on a class-action lawsuit brought against the US government by shareholders of AIG. Photo by Win McNamee/Getty Images
The crisis was too big for any one institution to take the fall, of course. A week after the AIG takeover, rumors began to circulate that Wachovia was suffering a silent run of high-net-worth depositors, and the requisite authorities began recruiting potential rescuers to acquire the bank. In the end only one came forward, and the term sheet was harsh. Citigroup would agree to "save" Wachovia at a price of $1 a share, CEO Vikram Pandit decided, if and only if the government agreed to guarantee $312 billion worth of the bank's assets—a move that would require the Federal Deposit Insurance Corporation to invoke a rarely used "systemic risk" clause in its charter.
But then Wells Fargo decided that Wachovia was worth $7 a share, with no government subsidies whatsoever, and made a counteroffer—incurring the wrath of Geithner, who recalled being "livid" at the news: "The United States government made a commitment," he said. "We can't act like we're a banana republic!"
Geithner wanted Citigroup to "save" Wachovia because Citi had hundreds of billions of dollars of worthless mortgages and mortgage-based products on its books—close to a trillion dollars in uninsured foreign deposits that could evaporate at any moment. It also had a 12-figure line of credit with the Fed and the Federal Home Loan Bank and literally thousands of lobbyists, publicists, emissaries, and miscellaneous fixers—from former chairman Bob Rubin to Obama crisis headhunter Michael Froman, who simultaneously led a private-equity division at Citi and was in the president elect's transition team during the fall of 2008—on its payroll. It desperately needed Wachovia's massive deposit base to make it through the month, and it had the clout to make that happen. The day after Wells made its competing bid, one of Citi's lobbyists slipped into the second draft of TARP legislation a line promising "exclusivity" to institutions recruited to rescue failing banks. The bill passed that evening, and the bank used the statute to get a judge to block the deal the next day.
With the help of celebrity lawyer David Boies, Wachovia and Wells ultimately prevailed, in part because the judge Citi persuaded to issue the restraining order had been a New York state judge and TARP was a federal statute. A week later, Citi got the first of three extra bailouts, in the form of one of the Treasury Department's "mandatory" capital injections, in an episode that later led FDIC chairman Sheila Bair to wonder whether the entire production was simply an elaborate propaganda campaign to bail out Citigroup with minimal public embarrassment.
"How much of the decision making was being driven through the prism of the special needs of that one, politically connected institution?" Bair asked later. "Were we throwing trillions of dollars at all of the banks to camouflage its problems? Were the others really in danger of failing? Or were we just softening the damage to their bottom lines through cheap capital and debt guarantees?"
If there is one person who is now positioned to tell us what the point of all this shameless flimflamming was, it is Boies, who is now arguing Hank Greenberg's case against the government. But while his meticulous reconstruction of events makes an irrefutable case that Geithner et al. acted intentionally, it makes no attempt to divine what their intentions ultimately were. If they were focused on saving Citigroup at any cost, then a $22 billion windfall for Goldman Sachs might have been one of those costs, but who knows. In the end it was all grotesquely profitable for Wall Street, just as the outcome of the Iraq War was great for war profiteers and the passage of time is generally great for the wealthy. But given how little popular opinion matters to the people who concoct these schemes, what warranted the elaborate disinformation campaign? In the end I think they just like fucking with us.
Follow Moe Tkacik on Twitter.
In my first Midori post, I described how safety was the foundation of everything we did. I mentioned that we built an operating system out of safe code, and yet stayed competetive with operating systems like Windows and Linux written in C and C++. In many ways, system architecture played a key role, and I will continue discussing how in future posts. But, at the foundation, an optimizing compiler that often eeked out native code performance from otherwise "managed", type- and memory-safe code, was one of our most important weapons. In this post, I'll describe some key insights and techniques that were essential to our success.
When people think of C#, Java, and related languages, they usually think of Just-In-Time (JIT) compilation. Especially back in the mid-2000s when Midori began. But Midori was different, using more C++-like Ahead-Of-Time (AOT) compilation from the outset.
AOT compiling managed, garbage collected code presents some unique challenges compared to C and C++. As a result, many AOT efforts don't achieve parity with their native counterparts. .NET's NGEN technology is a good example of this. In fact, most efforts in .NET have exclusively targeted startup time; this is clearly a key metric, but when you're building an operating system and everything on top, startup time just barely scratches the surface.
Over the course of 8 years, we were able to significantly narrow the gap between our version of C# and classical C/C++ systems, to the point where basic code quality, in both size of speed dimensions, was seldom the deciding factor when comparing Midori's performance to existing workloads. In fact, something counter-intuitive happened. The ability to co-design the language, runtime, frameworks, operating system, and the compiler -- making tradeoffs in one area to gain advantages in other areas -- gave the compiler far more symbolic information than it ever had before about the program's semantics and, so, I dare say, was able to exceed C and C++ performance in a non-trivial number of situations.
Before diving deep, I have to put in a reminder. The architectural decisions -- like Async Everywhere and Zero-Copy IO (coming soon) -- had more to do with us narrowing the gap at a "whole system" level. Especially the less GC-hungry way we wrote systems code. But the foundation of a highly optimizing compiler, that knew about and took advantage of safety, was essential to our results.
I would also be remiss if I didn't point out that the world has made considerable inroads in this area alongside us. Go has straddled an elegant line between systems performance and safety. Rust is just plain awesome. The .NET Native and, related, Android Runtime projects have brought a nice taste of AOT to C# and Java in a more limited way, as a "silent" optimization technique to avoid mobile application lag caused by JITting. Lately, we've been working on bringing AOT to a broader .NET setting with the CoreRT project. Through that effort I hope we can bring some of the lessons learned below to a real-world setting. Due to the delicate balance around breaking changes it remains to be seen how far we can go. It took us years to get everything working harmoniously, measured in man-decades, however, so this transfer of knowledge will take time.
First thing's first. Let's quickly recap: What's the difference between native and managed code, anyway?
I despise the false dichotomy "native and managed," so I must apologize for using it. After reading this article, I hope to have convinced you that it's a continuum. C++ is safer these days than ever before, and likewise, C# performant. It's amusing how many of these lessons apply directly to the work my team is doing on Safe C++ these days.
So let's begin by considering what's the same.
All the basic dragon book topics apply to managed as much as they do native code.
In general, compiling code is a balancing act between, on one hand emitting the most efficient instruction sequences for the target architecture, to execute the program quickly; and on the other hand emitting the smallest encoding of instructions for the target architecture, to store the program compactly and effectively use the memory system on the target device. Countless knobs exist on your favorite compiler to dial between the two based on your scenario. On mobile, you probably want smaller code, whereas on a multimedia workstation, you probably want the fastest.
The choice of managed code doesn't change any of this. You still want the same flexibility. And the techniques you'd use to achieve this in a C or C++ compiler are by and large the same as what you use for a safe language.
You need a great inliner. You want common subexpression elimination (CSE), constant propagation and folding, strength reduction, and an excellent loop optimizer. These days, you probably want to use static single assignment form (SSA), and some unique SSA optimizations like global value numbering (although you need to be careful about working set and compiler throughput when using SSA everywhere). You will need specialized machine dependent optimizers for the target architectures that are important to you, including register allocators. You'll eventually want a global analyzer that does interprocedural optimizations, link-time code-generation to extend those interprocedural optimizations across passes, a vectorizer for modern processors (SSE, NEON, AVX, etc.), and most definitely profile guided optimizations (PGO) to inform all of the above based on real-world scenarios.
Although having a safe language can throw some interesting curveballs your way that are unique and interesting -- which I'll cover below -- you'll need all of the standard optimizing compiler things.
I hate to say it, but doing great at all of these things is "table stakes." Back in the mid-2000s, we had to write everything by hand. Thankfully, these days you can get an awesome off-the-shell optimizing compiler like LLVM that has most of these things already battle tested, ready to go, and ready for you to help improve.
But, of course, there are differences. Many. This article wouldn't be very interesting otherwise.
The differences are more about what "shapes" you can expect to be different in the code and data structures thrown at the optimizer. These shapes come in the form of different instruction sequences, logical operations in the code that wouldn't exist in the C++ equivalent (like more bounds checking), data structure layout differences (like extra object headers or interface tables), and, in most cases, a larger quantity of supporting runtime data structures.
Objects have "more to them" in most managed languages, compared to frugal data types in, say, C. (Note that C++ data structures are not nearly as frugal as you might imagine, and are probably closer to C# than your gut tells you.) In Java, every object has a vtable pointer in its header. In C#, most do, although structs do not. The GC can impose extra layout restrictions, such as padding and a couple words to do its book-keeping. Note that none of this is really specific to managed languages -- C and C++ allocators can inject their own words too, and of course, many C++ objects also carry vtables -- however it's fair to say that most C and C++ implementations tend to be more economical in these areas. In most cases, for cultural reasons more than hard technical ones. Add up a few thousand objects in a heap, especially when your system is built of many small processes with isolated heaps, like Midori, and it adds up quickly.
In Java, you've got a lot more virtual dispatch, because methods are virtual by default. In C#, thankfully, methods are non-virtual by default. (We even made classes sealed by default.) Too much virtual dispatch can totally screw inlining which is a critical optimization to have for small functions. In managed languages you tend to have more small functions for two reasons: 1) properties, and 2) higher level programmers tend to over-use abstraction.
Although it's seldom described this formally, there's an "ABI" (Application Binary Interface) that governs interactions between code and the runtime. The ABI is where the rubber meets the road. It's where things like calling conventions, exception handling, and, most notably, the GC manifest in machine code. This is not unique to managed code! C++ has a "runtime" and therfore an ABI too. It's just that it's primarily composed of headers, libraries like allocators, and so on, that are more transparently linked into a program than with classical C# and Java virtual machines, where a runtime is non-negotiable (and in the JIT case, fairly heavy-handed). Thinking of it this way has been helpful to me, because the isomorphisms with C+ suddenly become immediately apparent.
The real biggie is array bounds checks. A traditional approach is to check that the index is within the bounds of an array before accessing it, either for laoding or storing. That's an extra field fetch, compare, and conditional branch. Branch prediction these days is quite good, however it's just plain physics that if you do more work, you're going to pay for it. Interestingly, the work we're doing with C++'s array_view<T> incurs all these same costs.
Related to this, there can be null checks where they didn't exist in C++. If you perform a method dispatch on a null object pointer in C++, for example, you end up running the function anyway. If that function tries to access this, it's bound to AV, but in Java and .NET, the compiler is required (per specification) to explicitly check and throw an exception in these cases, before the call even occurs. These little branches can add up too. We eradicated such checks in favor of C++ semantics in optimized builds.
In Midori, we compiled with overflow checking on by default. This is different from stock C#, where you must explicitly pass the /checked flag for this behavior. In our experience, the number of surprising overflows that were caught, and unintended, was well worth the inconvenience and cost. But it did mean that our compiler needed to get really good at understanding how to eliminate unnecessary ones.
Static variables are very expensive in Java and .NET. Way more than you'd expect. They are mutable and so cannot be stored in the readonly segment of an image where they are shared across processes. And my goodness, the amount of lazy-initialization checking that gets injected into the resulting source code is beyond belief. Switching from preciseinit to beforefieldinit semantics in .NET helps a little bit, since the checks needn't happen on every access to a static member -- just accesses to the static variable in question -- but it's still disgusting compared to a carefully crafted C program with a mixture of constant and intentional global initialization.
The final major area is specific to .NET: structs. Although structs help to alleviate GC pressure and hence are a good thing for most programs, they also carry some subtle problems. The CLI specifies surprising behavior around their initialization, for example. Namely if an exception happens during construction, the struct slot must remain zero- initialized. The result is that most compilers make defensive copies. Another example is that the compiler must make a defensive copy anytime you call a function on a readonly struct. It's pretty common for structs to be copied all over the place which, when you're counting cycles, hurts, especially since it often means time spent in memcpy. We had a lot of techniques for addressing this and, funny enough, I'm pretty sure when all was said and done, our code quality here was better than C++'s, given all of its RAII, copy constructor, destructor, and so on, penalties.
Our architecture involved three major components:
The similarities here with Swift's compiler design, particularly SIL, are evident. The .NET Native project also mirrors this architecture somewhat. Frankly, most AOT compilers for high level languages do.
In most places, the compiler's internal representation leveraged static single assignment form (SSA). SSA was preserved until very late in the compilation. This facilitated and improved the use of many of the classical compiler optimizations mentioned earlier.
The goals of this architecture included:
Finally, a brief warning. We tried lots of stuff. I can't remember it all. Both Bartok and Phoenix existed for years before I even got involved in them. Bartok was a hotbed of research on managed languages -- ranging from optimizations to GC to software transactional memory -- and Phoenix was meant to replace the shipping Visual C++ compiler. So, anyway, there's no way I can tell the full story. But I'll do my best.
Let's go deep on a few specific areas of classical compiler optimizations, extended to cover safe code.
C# arrays are bounds checked. So were ours. Although it is important to eliminate superfluous bounds checks in regular C# code, it was even more so in our case, because even the lowest layers of the system used bounds checked arrays. For example, where in the bowels of the Windows or Linux kernel you'd see an int*, in Midori you'd see an int[].
To see what a bounds check looks like, consider a simple example:
var a = new int[100];
for (int i = 0; i < 100; i++) {
... a[i] ...;
}
Here's is an example of the resulting machine code for the inner loop array access, with a bounds check:
; First, put the array length into EAX:
3B15: 8B 41 08 mov eax,dword ptr [rcx+8]
; If EDX >= EAX, access is out of bounds; jump to error:
3B18: 3B D0 cmp edx,eax
3B1A: 73 0C jae 3B28
; Otherwise, access is OK; compute element's address, and assign:
3B1C: 48 63 C2 movsxd rax,edx
3B1F: 8B 44 81 10 mov dword ptr [rcx+rax*4+10h],r8d
; ...
; The error handler; just call a runtime helper that throws:
3B28: E8 03 E5 FF FF call 2030
If you're doing this bookkeeping on every loop iteration, you won't get very tight loop code. And you're certianly not going to have any hope of vectorizing it. So, we spent a lot of time and energy trying to eliminate such checks.
In the above example, it's obvious to a human that no bounds checking is necessary. To a compiler, however, the analysis isn't quite so simple. It needs to prove all sorts of facts about ranges. It also needs to know that a isn't aliased and somehow modified during the loop body. It's surprising how hard this problem quickly becomes.
Our system had multiple layers of bounds check eliminations.
First it's important to note that CIL severely constraints an optimizer by being precise in certain areas. For example, accessing an array out of bounds throws an IndexOutOfRangeException, similar to Java's ArrayOutOfBoundsException. And the CIL specifies that it shall do so at precisely the exception that threw it. As we will see later on, our error model was more relaxed. It was based fail-fast and permitted code motion that led to inevitable failures happening "sooner" than they would have otherwise. Without this, our hands would have been tied for much of what I'm about to discuss.
At the highest level, in Bartok, the IR is still relatively close to the program input. So, some simple patterns could be matched and eliminated. Before lowering further, the ABCD algorithm -- a straightforward value range analysis based on SSA -- then ran to eliminate even more common patterns using a more principled approach than pattern matching. We were also able to leverage ABCD in the global analysis phase too, thanks to inter-procedural length and control flow fact propagation.
Next up, the Phoenix Loop Optimizer got its hands on things. This layer did all sorts of loop optimizations and, most relevant to this section, range analysis. For example:
This sort of principled analysis was more capable than what was shown earlier. For example, there are ways to write the earlier loop that can easily "trick" the more basic techniques discussed earlier:
var a = new int[100];
// Trick #1: use the length instead of constant.
for (int i = 0; i < a.length; i++) {
a[i] = i;
}
// Trick #2: start counting at 1.
for (int i = 1; i <= a.length; i++) {
a[i-1] = i-1;
}
// Trick #3: count backwards.
for (int i = a.length - 1; i >= 0; i--) {
a[i] = i;
}
// Trick #4: don't use a for loop at all.
int i = 0;
next:
if (i < a.length) {
a[i] = i;
i++;
goto next;
}
You get the point. Clearly at some point you can screw the optimizer's ability to do anything, especially if you start doing virtual dispatch inside the loop body, where aliasing information is lost. And obviously, things get more difficult when the array length isn't known statically, as in the above example of 100. All is not lost, however, if you can prove relationships between the loop bounds and the array. Much of this analysis requires special knowledge of the fact that array lengths in C# are immutable.
At the end of the day, doing a good job at optimizing here is the difference between this:
; Initialize induction variable to 0:
3D45: 33 C0 xor eax,eax
; Put bounds into EDX:
3D58: 8B 51 08 mov edx,dword ptr [rcx+8]
; Check that EAX is still within bounds; jump if not:
3D5B: 3B C2 cmp eax,edx
3D5D: 73 13 jae 3D72
; Compute the element address and store into it:
3D5F: 48 63 D0 movsxd rdx,eax
3D62: 89 44 91 10 mov dword ptr [rcx+rdx*4+10h],eax
; Increment the loop induction variable:
3D66: FF C0 inc eax
; If still < 100, then jump back to the loop beginning:
3D68: 83 F8 64 cmp eax,64h
3D6B: 7C EB jl 3D58
; ...
; Error routine:
3D72: E8 B9 E2 FF FF call 2030
And the following, completely optimized, bounds check free, loop:
; Initialize induction variable to 0:
3D95: 33 C0 xor eax,eax
; Compute the element address and store into it:
3D97: 48 63 D0 movsxd rdx,eax
3D9A: 89 04 91 mov dword ptr [rcx+rdx*4],eax
; Increment the loop induction variable:
3D9D: FF C0 inc eax
; If still < 100, then jump back to the loop beginning:
3D9F: 83 F8 64 cmp eax,64h
3DA2: 7C F3 jl 3D97
It's amusing that I'm now suffering deja vu as we go through this same exercise with C++'s new array_view<T> type. Sometimes I joke with my ex-Midori colleagues that we're destined to repeat ourselves, slowly and patiently, over the course of the next 10 years. I know that sounds arrogant. But I have this feeling on almost a daily basis.
As mentioned earlier, in Midori we compiled with checked arithmetic by default (by way of C#'s /checked flag). This eliminated classes of errors where developers didn't anticipate, and therefore code correctly for, overflows. Of course, we kept the explicit checked and unchecked scoping constructs, to override the defaults when appropriate, but this was preferable because a programmer declared her intent.
Anyway, as you might expect, this can reduce code quality too.
For comparison, imagine we're adding two variables:
int x = ...;
int y = ...;
int z = x + y;
Now imagine x is in ECX and y is in EDX. Here is a standard unchecked add operation:
Or, if you want to get fancy, one that uses the LEA instruction to also store the result in the EAX register using a single instruction, as many modern compilers might do:
8D 04 11 lea eax,[rcx+rdx]
Well, here's the equivalent code with a bounds check inserted into it:
3A65: 8B C1 mov eax,ecx
3A67: 03 C2 add eax,edx
3A69: 70 05 jo 3A70
; ...
3A70: E8 B3 E5 FF FF call 2028
More of those damn conditional jumps (JO) with error handling routines (CALL 2028).
It turns out a lot of the analysis mentioned earlier that goes into proving bounds checks redundant also apply to proving that overflow checks are redundant. It's all about proving facts about ranges. For example, if you can prove that some check is dominated by some earlier check, and that furthermore that earlier check is a superset of the later check, then the later check is unnecessary. If the opposite is true -- that is, the earlier check is a subset of the later check, then if the subsequent block postdominates the earlier one, you might move the stronger check to earlier in the program.
Another common pattern is that the same, or similar, arithmetic operation happens multiple times near one another:
int p = r * 32 + 64;
int q = r * 32 + 64 - 16;
It is obvious that, if the p assignment didn't overflow, then the q one won't either.
There's another magical phenomenon that happens in real world code a lot. It's common to have bounds checks and arithmetic checks in the same neighborhood. Imagine some code that reads a bunch of values from an array:
int data0 = data[dataOffset + (DATA_SIZE * 0)];
int data1 = data[dataOffset + (DATA_SIZE * 1)];
int data2 = data[dataOffset + (DATA_SIZE * 2)];
int data3 = data[dataOffset + (DATA_SIZE * 3)];
.. and so on ...
Well C# arrays cannot have negative bounds. If a compiler knows that DATA_SIZE is sufficiently small that an overflowed computation won't wrap around past 0, then it can eliminate the range check in favor of the bounds check.
There are many other patterns and special cases you can cover. But the above demonstrates the power of a really good range optimizer that is integrated with loops optimization. It can cover a wide array of scenarios, array bounds and arithmetic operations included. It takes a lot of work, but it's worth it in the end.
For the most part, inlining is the same as with true native code. And just as important. Often more important, due to C# developers' tendency to write lots of little methods (like property accessors). Because of many of the topics throughout this article, getting small code can be more difficult than in C++ -- more branches, more checks, etc. -- and so, in practice, most managed code compilers inline a lot less than native code compilers, or at least need to be tuned very differently. This can actually make or break performance.
There are also areas of habitual bloat. The way lambdas are encoded in MSIL is unintelligable to a naive backend compiler, unless it reverse engineers that fact. For example, we had an optimization that took this code:
void A(Action a) {
a();
}
void B() {
int x = 42;
A(() => x++);
...
}
and, after inlining, was able to turn B into just:
void B() {
int x = 43;
...
}
That Action argument to A is a lambda and, if you know how the C# compiler encodes lambdas in MSIL, you'll appreciate how difficult this trick was. For example, here is the code for B:
.method private hidebysig instance void
B() cil managed
{
// Code size 36 (0x24)
.maxstack 3
.locals init (class P/'<>c__DisplayClass1' V_0)
IL_0000: newobj instance void P/'<>c__DisplayClass1'::.ctor()
IL_0005: stloc.0
IL_0006: nop
IL_0007: ldloc.0
IL_0008: ldc.i4.s 42
IL_000a: stfld int32 P/'<>c__DisplayClass1'::x
IL_000f: ldarg.0
IL_0010: ldloc.0
IL_0011: ldftn instance void P/'<>c__DisplayClass1'::'<B>b__0'()
IL_0017: newobj instance void [mscorlib]System.Action::.ctor(object,
native int)
IL_001c: call instance void P::A(class [mscorlib]System.Action)
IL_0021: nop
IL_0022: nop
IL_0023: ret
}
To get the magic result required constant propagating the ldftn, recognizing how delegate construction works (IL_0017), leveraging that information to inline B and eliminate the lambda/delegate altogether, and then, again mostly through constant propagation, folding the arithmetic into the constant 42 initialization of x. I always found it elegant that this "fell out" of a natural composition of multiple optimizations with separate concerns.
As with native code, profile guided optimization made our inlining decisions far more effective.
CLI structs are almost just like C structs. Except they're not. The CLI imposes some semantics that incur overheads. These overheads almost always manifest as excessive copying. Even worse, these copies are usually hidden from your program. It's worth noting, because of copy constructors and destructors, C++ also has some real issues here, often even worse than what I'm about to describe.
Perhaps the most annoying is that initializing a struct the CLI way requires a defensive copy. For example, consider this program, where the initialzer for S throws an exception:
class Program {
static void Main() {
S s = new S();
try {
s = new S(42);
}
catch {
System.Console.WriteLine(s.value);
}
}
}
struct S {
public int value;
public S(int value) {
this.value = value;
throw new System.Exception("Boom");
}
}
The program behavior here has to be that the value 0 is written to the console. In practice, that means that the assignment operation s = new S(42) must first create a new S-typed slot on the stack, construct it, and then and only then copy the value back over the s variable. For single-int structs like this one, that's not a huge deal. For large structs, that means resorting to memcpy. In Midori, we knew what methods could throw, and which could not, thanks to our error model (more later), which meant we could avoid this overhead in nearly all cases.
Another annoying one is the following:
struct S {
// ...
public int Value { get { return this.value; } }
}
static readonly S s = new S();
Every single time we read from s.Value:
we are going to get a local copy. This one's actually visible in the MSIL. This is without readonly:
ldsflda valuetype S Program::s
call instance int32 S::get_Value()
And this is with it:
ldsfld valuetype S Program::s
stloc.0
ldloca.s V_0
call instance int32 S::get_Value()
Notice that the compiler elected to use ldsfld followed by lodloca.s, rather than loading the address directly, by way of ldsflda in the first example. The resulting machine code is even nastier. I also can't pass the struct around by-reference which, as I mention later on, requires copying it and again can be problematic.
We solved this in Midori because our compiler knew about methods that didn't mutate members. All statics were immutable to begin with, so the above s wouldn't need defensive copies. Alternatively, or in addition to this, the struct could have beem declared as immutable, as follows:
immutable struct S {
// As above ...
}
Or because all static values were immutable anyway. Alternatively, the properties or methods in question could have been annotated as readable meaning that they couldn't trigger mutations and hence didn't require defensive copies.
I mentioned by-reference passing. In C++, developers know to pass large structures by-reference, either using * or &, to avoid excessive copying. We got in the habit of doing the same. For example, we had in parameters, as so:
void M(in ReallyBigStruct s) {
// Read, but don't assign to, s ...
}
I'll admit we probably took this to an extreme, to the point where our APIs suffered. If I could do it all over again, I'd go back and eliminate the fundamental distinction between class and struct in C#. It turns out, pointers aren't that bad after all, and for systems code you really do want to deeply understand the distinction between "near" (value) and "far" (pointer). We did implement what amounted to C++ references in C#, which helped, but not enough. More on this in my upcoming deep dive on our programming language.
We pushed hard on code size. Even more than some C++ compilers I know.
A generic instantiation is just a fancy copy-and-paste of code with some substitutions. Quite simply, that means an explosion of code for the compiler to process, compared to what the developer actually wrote. I've covered many of the performance challenges with generics in the past. A major problem there is the transitive closure problem. .NET's straightforward-looking List<T> class actually creates 28 types in its transitive closure! And that's not even speaking to all the methods in each type. Generics are a quick way to explode code size.
I never forgot the day I refactored our LINQ implementation. Unlike in .NET, which uses extension methods, we made all LINQ operations instance methods on the base-most class in our collection type hierarchy. That meant 100-ish nested classes, one for each LINQ operation, for every single collection instantiated! Refactoring this was an easy way for me to save over 100MB of code size across the entire Midori "workstation" operating system image. Yes, 100MB!
We learned to be more thoughtful about our use of generics. For example, types nested inside an outer generic are usually not good ideas. We also aggressively shared generic instantiations, even more than what the CLR does. Namely, we shared value type generics, where the GC pointers were at the same locations. So, for example, given a struct S:
we would share the same code representation of List<int> with List<S>. And, similarly, given:
struct S {
object A;
int B;
object C;
}
struct T {
object D;
int E;
object F;
}
we would share instantiations between List<S> and List<T>.
You might not realize this, but C# emits IL that ensures structs have sequential layout:
.class private sequential ansi sealed beforefieldinit S
extends [mscorlib]System.ValueType
{
...
}
As a result, we couldn't share List<S> and List<T> with some hypothetical List<U>:
struct U {
int G;
object H;
object I;
}
For this, among other reasons -- like giving the compiler more flexibility around packing, cache alignment, and so on -- we made structs auto by default in our language. Really, sequential only matters if you're doing unsafe code, which, in our programming model, wasn't even legal.
We did not support reflection in Midori. In principle, we had plans to do it eventually, as a purely opt-in feature. In practice, we never needed it. What we found is that code generation was always a more suitable solution. We shaved off at least 30% of the best case C# image size by doing this. Significantly more if you factor in systems where the full MSIL is retained, as is usually the case, even for NGen and .NET AOT solutions.
In fact, we removed significant pieces of System.Type too. No Assembly, no BaseType, and yes, even no FullName. The .NET Framework's mscorlib.dll contains about 100KB of just type names. Sure, names are useful, but our eventing framework leveraged code generation to produce just those you actually needed to be around at runtime.
At some point, we realized 40% of our image sizes were vtables. We kept pounding on this one relentlessly, and, after all of that, we still had plenty of headroom for improvements.
Each vtable consumes image space to hold pointers to the virtual functions used in dispatch, and of course has a runtime representation. Each object with a vtable also has a vtable pointer embedded within it. So, if you care about size (both image and runtime), you are going to care about vtables.
In C++, you only get a vtable if you use virtual inheritance. In languages like C# and Java, you get them even if you didn't want them. In C#, at least, you can use a struct type to elide them. I actually love this aspect of Go, where you get a virtual dispatch-like thing, via interfaces, without needing to pay for vtables on every type; you only pay for what you use, at the point of coercing something to an interface.
Another vtable problem in C# is that all objects inherit three virtuals from System.Object: Equals, GetHashCode, and ToString. Besides the point that these generally don't do the right thing in the right way anyways -- Equals requires reflection to work on value types, GetHashCode is nondeterministic and stamps the object header (or sync- block; more on that later), and ToString doesn't offer formatting and localization controls -- they also bloat every vtable by three slots. This may not sound like much, but it's certainly more than C++ which has no such overhead.
The main source of our remaining woes here was the assumption in C#, and frankly most OOP languages like C++ and Java, that RTTI is always available for downcasts. This was particularly painful with generics, for all of the above reasons. Although we aggressively shared instantiations, we could never quite fully fold together the type structures for these guys, even though disparate instantiations tended to be identical, or at least extraordinarily similar. If I could do it all over agan, I'd banish RTTI. In 90% of the cases, type discriminated unions or pattern matching are more appropriate solutions anyway.
I've mentioned profile guided optimization (PGO) already. This was a critical element to "go that last mile" after mostly everything else in this article had been made competetive. This gave our browser program boosts in the neighborhood of 30-40% on benchmarks like SunSpider and Octane.
Most of what went into PGO was similar to classical native profilers, with two big differences.
First, we tought PGO about many of the unique optimizations listed throughout this article, such as asynchronous stack probing, generics instantiations, lambdas, and more. As with many things, we could have gone on forever here.
Second, we experimented with sample profiling, in addition to the ordinary instrumented profiling. This is much nicer from a developer perspective -- they don't need two builds -- and also lets you collect counts from real, live running systems in the data center. A good example of what's possible is outlined in this Google-Wide Profiling (GWP) paper.
The basics described above were all important. But a number of even more impactful areas required deeper architectural co-design and co-evolution with the language, runtime, framework, and operating system itself. I've written about the immense benefits of this sort of "whole system" approach before. It was kind of magical.
Midori was garbage collected through-and-through. This was a key element of our overall model's safety and productivity. In fact, at one point, we had 11 distinct collectors, each with its own unique characteristics. (For instance, see this study.) We had some ways to combat the usual problems, like long pause times. I'll go through those in a future post, however. For now, let's stick to the realm of code quality.
The first top-level decision is: conservative or precise? A conserative collector is easier to wedge into an existing system, however it can cause troubles in certain areas. It often needs to scan more of the heap to get the same job done. And it can falsely keep objects alive. We felt both were unacceptable for a systems programming environment. It was an easy, quick decision: we sought precision.
Precision costs you something in the code generators, however. A precise collector needs to get instructions where to find its root set. That root set includes field offsets in data structures in the heap, and also places on the stack or, even in some cases, registers. It needs to find these so that it doesn't miss an object and erroneously collect it or fail to adjust a pointer during a relocation, both of which would lead to memory safety problems. There was no magic trick to making this efficient other than close integration between runtime and code generator, and being thoughtful.
This brings up the topic of cooperative versus preemptive, and the notion of GC safe-points. A GC operating in cooperative mode will only collect when threads have reached so-called "safe-points." A GC operating in preemptive mode, on the other hand, is free to stop threads in their tracks, through preemption and thread suspension, so that it may force a collection. In general, preemptive requires more bookkeeping, because the roots must be identifiable at more places, including things that have spilled into registers. It also makes certain low-level code difficult to write, of the ilk you'll probably find in an operating system's kernel, because objects are subject to movement between arbitrary instructions. It's difficult to reason about. (See this file, and its associated uses in the CLR codebase, if you don't believe me.) As a result, we used cooperative mode as our default. We experimented with automatic safe-point probes inserted by the compiler, for example on loop back-edges, but opted to bank the code quality instead. It did mean GC "livelock" was possible, but in practice we seldom ran into this.
We used a generational collector. This has the advantage of reducing pause times because less of the heap needs to be inspected upon a given collection. It does come with one disadvantage from the code generator's perspective, which is the need to insert write barriers into the code. If an older generation object ever points back at a younger generation object, then the collector -- which would have normally preferred to limit its scope to younger generations -- must know to look at the older ones too. Otherwise, it might miss something.
Write barriers show up as extra instructions after certain writes; e.g., note the call:
48 8D 49 08 lea rcx,[rcx+8]
E8 7A E5 FF FF call 0000064488002028
That barrier simply updates an entry in the card table, so the GC knows to look at that segment the next time it scans the heap. Most of the time this ends up as inlined assembly code, however it depends on the particulars of the situation. See this code for an example of what this looks like for the CLR on x64.
It's difficult for the compiler to optimize these away because the need for write barriers is "temporal" in nature. We did aggressively eliminate them for stack allocated objects, however. And it's possible to write, or transform code, into less barrier hungry styles. For example, consider two ways of writing the same API:
bool Test(out object o);
object Test(out bool b);
In the resulting Test method body, you will find a write barrier in the former, but not the latter. Why? Because the former is writing a heap object reference (of type object), and the compiler has no idea, when analyzing this method in isolation, whether that write is to another heap object. It must be conservative in its analysis and assume the worst. The latter, of course, has no such problem, because a bool isn't something the GC needs to scan.
Another aspect of GC that impacts code quality is the optional presence of more heavyweight concurrent read and write barriers, when using concurrent collection. A concurrent GC does some collection activities concurrent with the user program making forward progress. This is often a good use of multicore processors and it can reduce pause times and help user code make more forward progress over a given period of time.
There are many challenges with building a concurrent GC, however one is that the cost of the resulting barriers is high. The original concurrent GC by Henry Baker was a copying GC and had the notion of "old" versus "new" space. All reads and writes had to be checked and, anything operation against the old space had to be forwarded to the new space. Subsequent research for the DEC Firefly used hardware memory protection to reduce the cost, but the faulting cases were still exceedingly expensive. And, worst of all, access times to the heap were unpredictable. There has been a lot of good research into solving this problem, however we abandoned copying.
Instead, we used a concurrent mark-sweep compacting collector. This means only write barriers are needed under normal program execution, however some code was cloned so that read barriers were present when programs ran in the presence of object movement. Our primary GC guy's research was published, so you can read all about it. The CLR also has a concurrent collector, but it's not quite as good. It uses copying to collect the youngest generation, mark-sweep for the older ones, and the mark phase is parallelized. There are unfortunately a few conditions that can lead to sequential pauses (think of this like a big "lock"), sometimes over 10 milliseconds: 1) all threads must be halted and scanned, an operation that is bounded only by the number of threads and the size of their stacks; 2) copying the youngest generation is bounded only by the size of that generation (thankfully, in normal configurations, this is small); and 3) under worst case conditions, compaction and defragmentation, even of the oldest generation, can happen.
The basic model to start with is static linking. In this model, you compile everything into a single executable. The benefits of this are obvious: it's simple, easy to comprehend, conceptually straightforward to service, and less work for the entire compiler toolchain. Honestly, given the move to Docker containers as the unit of servicing, this model makes more and more sense by the day. But at some point, for an entire operating system, you'll want separate compilation. Not just because compile times can get quite long when statically linking an entire operating system, but also because the working set and footprint of the resulting processes will be bloated with significant duplication.
Separately compiling object oriented APIs is hard. To be honest, few people have actually gotten it to work. Problems include the fragile base class problem, which is a real killer for version resilient libraries. As a result, most real systems use a dumbed down at the boundary between components. This is why Windows, for example, has historically used flat C Win32 APIs and, even in the shift to more object orientation via WinRT, uses COM underneath it all. At some runtime expense, the ObjectiveC runtime addressed this challenge. As with most things in computer science, virtually all problems can be solved with an extra level of indirection; this one can be too.
The design pivot we took in Midori was that whole processes were sealed. There was no dynamic loading, so nothing that looked like classical DLLs or SOs. For those scenarios, we used the Asynchronous Everything programming model, which made it easy to dynamically connect to and use separately compiled and versioned processes.
We did, however, want separately compiled binaries, purely as a developer productivity and code sharing (working set) play. Well, I lied. What we ended up with was incrementally compiled binaries, where a change in a root node triggered a cascading recompilation of its dependencies. But for leaf nodes, such as applications, life was beautiful. Over time, we got smarter in the toolchain by understanding precisely which sorts of changes could trigger cascading invaliation of images. A function that was known to never have been inlined across modules, for example, could have its implementation -- but not its signature -- changed, without needing to trigger a rebuild. This is similar to the distinction between headers and objects in a classical C/C++ compilation model.
Our compilation model was very similar to C++'s, in that there was static and dynamic linking. The runtime model, of course, was quite different. We also had the notion of "library groups," which let us cluster multiple logically distinct, but related, libraries into a single physical binary. This let us do more aggressive inter-module optimizations like inlining, devirtualization, async stack optimizations, and more.
That brings me to generics. They throw a wrench into everything.
The problem is, unless you implement an erasure model -- which utterly stinks performance-wise due to boxing allocations, indirections, or both -- there's no way for you to possibly pre-instantiate all possible versions of the code ahead-of-time. For example, say you're providing a List<T>. How do you know whether folks using your library will want a List<int>, List<string>, or List<SomeStructYouveNeverHeardOf>?
Solutions abound:
Java uses #1 (in fact, erasure is baked into the language). Many ML compilers use #2. .NET's NGen compilation model is sort of a variant of #2, where things that can be trivially specialized are specialized, and everything else is JIT compiled. .NET Native doesn't yet have a solution to this problem, which means 3rd party libraries, separate compilation, and generics are a very big TBD. As with everything in Midori, we picked the hardest path, with the most upside, which meant #3. Actually I'm being a little glib; we had several ML compiler legends on the team, and #2 is fraught with peril; just dig a little into some papers on how hard (and clever) this can get. It's difficult to know a priori which instantiations are going to be performance critical to a program. My own experience trying to get C# code into the heart of Windows back in the Longhorn days also reinforced this; we didn't want JIT'ting and the rules for what generics you could and couldn't use in that world were so mind boggling they eventually led to greek formulas.
Anyway, Midori's approach turned out to be harder than it sounded at first.
Imagine you have a diamond. Library A exports a List<T> type, and libraries B and C both instantiate List<int>. A program D then consumes both B and C and maybe even passes List<T> objects returned from one to the other. How do we ensure that the versions of List<int> are compatible?
We called this problem the potentially multiply instantiated, or PMI for short, problem.
The CLR handles this problem by unifying the instantiations at runtime. All RTTI data structures, vtables, and whatnot, are built and/or aggressively patched at runtime. In Midori, on the other hand, we wanted all such data structures to be in readonly data segments and hence shareable across processes, wherever possible.
Again, everything can be solved with an indirection. But unlike solution #2 above, solution #3 permits you to stick instantiations only in the rare places where you need them. And for purposes of this one, that meant RTTI and accessing static variables of just those generic types that might have been subject to PMI. First, that affected a vast subset of code (versus #2 which generally affects even loading of instance fields). Second, it could be optimized away for instantiations that were known not to be PMI, by attaching state and operations to the existing generic dictionary that was gets passed around as a hidden argument already. And finally, because of all of this, it was pay for play.
But damn was it complex.
It's funny, but C++ RTTI for template instantiations actually suffers from many of the same problems. In fact, the Microsoft Visual C++ compiler resorts to a strcmp of the type names, to resolve diamond issues! (Thankfully there are well-known, more efficient ways to do this, which we are actively pursuing for the next release of VC++.)
Although I felt differently when first switching from Java to C#, Midori made me love that C# made methods non-virtual by default. I'm sure we would have had to change this otherwise. In fact, we went even further and made classes sealed by default, requiring that you explicitly mark them virtual if you wanted to facilitate subclasses.
Aggressive devirtualization, however, was key to good performance. Each virtual means an indirection. And more impactfully, a lost opportunity to inline (which for small functions is essential). We of course did global intra-module analysis to devirtualize, but also extended this across modules, using whole program compilation, when multiple binaries were grouped together into a library group.
Although our defaults were right, m experience with C# developers is that they go a little hog-wild with virtuals and overly abstract code. I think the ecosystem of APIs that exploded around highly polymorphic abstractions, like LINQ and Reactive Extensions, encouraged this and instilled lots of bad behavior. As you can guess, there wasn't very much of that floating around our codebase. A strong culture around identifying and trimming excessive fat helped keep this in check, via code reviews, benchmarks, and aggressive static analysis checking.
Interfaces were a challenge.
There are just some poorly designed, inefficient patterns in the .NET Framework. IEnumerator<T> requires two interface dispatches simply to extract the next item! Compare that to C++ iterators which can compile down a pointer increment plus dereference. Many of these problems could be addressed simply with better library designs. (Our final design for enumeration didn't even invole interfaces at all.)
Plus invoking a C# interface is tricky. Existing systems do not use pointer adjustment like C++ does so usually an interface dispatch requires a table search. First a level of indirection to get to the vtable, then another level to find the interface table for the interface in question. Some systems attempt to do callsite caching for monomorphic invocations; that is, caching the latest invocation in the hope that the same object kind passes through that callsite time and time again. This requires mutable stubs, however, not to mention an incredibly complex system of thunks and whatnot. In Midori, we never ever ever violated W^X; and we avoided mutable runtime data structures, because they inhibit sharing, both in terms of working set, but also amortizing TLB and data cache pressure.
Our solution took advantage of the memory ordering model earlier. We used so-called "fat" interface pointers. A fat interface pointer was two words: the first, a pointer to the object itself; the second, a pointer to the interface vtable for that object. This made conversion to interfaces slightly slower -- because the interface vtable lookup had to happen -- but for cases where you are invoking it one or more times, it came out a wash or ahead. Usually, significantly. Go does something like this, but it's slightly different for two reasons. First, they generate the interface tables on the fly, because interfaces are duck typed. Second, fat interface pointers are subject to tearing and hence can violate memory safety in Go, unlike Midori thanks to our strong concurrency model.
The finally challenge in this category was generic virtual methods, or GVMs. To cut to the chase, we banned them. Even if you NGen an image in .NET, all it takes is a call to the LINQ query a.Where(...).Select(...), and you're pulling in the JIT compiler. Even in .NET Native, there is considerable runtime data structure creation, lazily, when this happens. In short, there is no known way to AOT compile GVMs in a way that is efficient at runtime. So, we didn't even bother offering them. This was a slightly annoying limitation on the programming model but I'd have done it all over again thanks to the efficiencies that it bought us. It really is surprising how many GVMs are lurking in .NET.
I was astonished the day I learned that 10% of our code size was spent on static initialization checks.
Many people probably don't realize that the CLI specification offers two static initialization modes. There is the default mode and beforefieldinit. The default mode is the same as Java's. And it's horrible. The static initializer will be run just prior to accessing any static field on that type, any static method on that type, any instance or virtual method on that type (if it's a value type), or any constructor on that type. The "when" part doesn't matter as much as what it takes to make this happen; all of those places now need to be guarded with explicit lazy initialization checks in the resulting machine code!
The beforefieldinit relaxation is weaker. It guarantees the initializer will run sometime before actually accessing a static field on that type. This gives the compiler a lot of leeway in deciding on this placement. Thankfully the C# compiler will pick beforefieldinit automatically for you should you stick to using field initializers only. Most people don't realize the incredible cost of choosing instead to use a static constructor, however, especially for value types where suddenly all method calls now incur initialization guards. It's just the difference between:
struct S {
static int Field = 42;
}
and:
struct S {
static int Field;
static S() {
Field = 42;
}
}
Now imagine the struct has a property:
struct S {
// As above...
int InstanceField;
public int Property { get { return InstanceField; } }
}
Here's the machine code for Property if S has no static initializer, or uses beforefieldinit (automatically injected by C# in the the field initializer example above):
; The struct is one word; move its value into EAX, and return it:
8B C2 mov eax,edx
C3 ret
And here's what happens if you add a class constructor:
; Big enough to get a frame:
56 push rsi
48 83 EC 20 sub rsp,20h
; Load the field into ESI:
8B F2 mov esi,edx
; Load up the cctor's initialization state:
48 8D 0D 02 D6 FF FF lea rcx,[1560h]
48 8B 09 mov rcx,qword ptr [rcx]
BA 03 00 00 00 mov edx,3
; Invoke the conditional initialization helper:
E8 DD E0 FF FF call 2048
; Move the field from ESI into EAX, and return it:
8B C6 mov eax,esi
48 83 C4 20 add rsp,20h
5E pop rsi
On every property access!
Of course, all static members still incur these checks, even if beforefieldinit is applied.
Although C++ doesn't suffer this same problem, it does have mind-bending initialization ordering semantics. And, like C# statics, C++11 introduced thread-safe initialization, by way of the .
We virtually eliminated this entire mess in Midori.
I mentioned offhandedly earlier that Midori had no mutable statics. More accurately, we extended the notion of const to cover any kind of object. This meant that static values were evaluated at compile-time, written to the readonly segment of the resulting binary image, and shared across all processes. More importantly for code quality, all runtime initialization checks were removed, and all static accesses simply replaced with a constant address.
There were still mutable statics at the core of the system -- in the kernel, for example -- but these did not make their way up into user code. And because they were few and far between, we did not rely on the classical C#-style lazy initialization checks for them. They were manually initialized on system startup.
As I said earlier, a 10% reduction in code size, and lots of speed improvements. It's hard to know exactly how much saved this was than a standard C# program because by the time we made the change, developers were well aware of the problems and liberally applied our [BeforeFieldInit] attribute all over their types, to avoid some of the overheads. So the 10% number is actually a lower bound on the savings we realized throughout this journey.
I already wrote a lot about our async model. I won't rehash all of that here. I will reiterate one point: the compiler was key to making linked stacks work.
In a linked stacks model, the compiler needs to insert probes into the code that check for available stack space. In the event there isn't enough to perform some operation -- make a function call, dynamically allocate on the stack, etc. -- the compiler needs to arrange for a new link to get appended, and to switch to it. Mostly this amounts to some range checking, a conditional call to a runtime function, and patching up RSP. A probe looked something like:
; Check amount of stack space:
lea rax, [rsp-250h]
cmp rax, qword ptr gs:[0]
ja prolog
; If insufficient stack, link a new segment:
mov eax, 10029h
call ?g_LinkNewStackTrampoline
prolog:
; The real code goes here...
Needless to say, you want to probe as little as possible, for two reasons. First, they incur runtime expense. Second, they chew up code size. There are a few techniques we used to eliminate probes.
The compiler of course knew how to compute stack usage of functions. As a result, it could be smart about the amount of memory to probe for. We incorporated this knowledge into our global analyzer. We could coalesce checks after doing code motion and inlining. We hoisted checks out of loops. For the most part, we optimized for eliminating checks, sometimes at the expense of using a little more stack.
The most effective technique we used to eliminate probes was to run synchronous code on a classical stack, and to teach our compiler to elide probes altogether for them. This took advantage of our understanding of async in the type system. Switching between the classical stack and back again again amounted to twiddling RSP:
; Switch to the classical stack:
move rsp, qword ptr gs:[10h]
sub rsp, 20h
; Do some work (like interop w/ native C/C++ code)...
; Now switch back:
lea rsp, [rbp-50h]
I know Go abandoned linked stacks because of these switches. At first they were pretty bad for us, however after about a man year or two of effort, the switching time faded away into the sub-0.5% noise.
Midori's stance on safe concurrency had truly one amazing benefit: you get a sequentially consistent memory ordering model for free. You may wish to read that again. Free!
Why is this so? First, Midori's process model ensured single-threaded execution by default. Second, any fine-grained parallelism inside of a process was governed by a finite number of APIs, all of which were race-free. The lack of races meant we could inject a fence at fork and join points, selectively, without a developer needing to care or know.
Obviously this had incredible benefits to developer productivity. The fact that Midori programmers never got bitten by memory reordering problems was certainly one of my proudest outcomes of the project.
But it also meant the compiler was free to make more aggressive code motion optimizations, without any sacrifices to this highly productive programming model. In other words, we got the best of both worlds.
A select few kernel developers had to think about the memory ordering model of the underlying machine. These were the people implementing the async model itself. For that, we eliminated C#'s notion of volatile -- which is utterly broken anyway -- in favor of something more like C++ atomics. That model is quite nice for two reasons. First, what kind of fence you need is explicit for every read and write, where it actually matters. (ences affect the uses of a variable, not its declaration. Second, the explicit model tells the compiler more information about what optimizations can or cannot take place, again at a specific uses, where it matters most.
Our error model journey was a long one and will be the topic of a future post. In a nutshell, however, we experimented with two ends of the spectrum -- exceptions and return codes -- and lots of points in betweeen.
Here is what we found from a code quality perspective.
Return codes are nice because the type system tells you an error can happen. A developer is thus forced to deal with them (provided they don't ignore return values). Return codes are also simple, and require far less "runtime magic" than exceptions or related mechanisms like setjmp/longjmp. So, lots to like here.
From a code quality persective, however, return codes suck. They force you to execute instructions in hot paths that wouldn't have otherwise been executed, including when errors aren't even happening. You need to return a value from your function -- occupying register and/or stack space -- and callers need to perform branches to check the results. Granted, we hope that these are predicted correctly, but the reality is, you're just doing more work.
Untyped exceptions suck when you're trying to build a reliable system. Operating systems need to be reliable. Not knowing that there's a hidden control flow path when you're calling a function is, quite simply, unacceptable. They also require heavier weight runtime support to unwind stacks, search for handlers, and so on. It's also a bitch to model exceptional control flow in the compiler. (If you don't believe me, just read through this mail exchange. So, lots to hate here.
Typed exceptions -- I got used to not saying checked exceptions for fear of hitting Java nerves -- address some of these shortcomings, but come with their own challenges. Again, I'll save detailed analysis for my future post.
From a code quality perspective, exceptions can be nice. First, you can organize code segments so that the "cold" handlers aren't dirtying your ICACHE on successful pathways. Second, you don't need to perform any extra work during the normal calling convention. There's no wrapping of values -- so no extra register or stack pressure -- and there's no branching in callers. There can be some downsides to exceptions, however. In an untyped model, you must assume every function can throw, which obviously inhibits your ability to move code around.
Our model ended up being a hybrid of two things:
I'd say the ratio of fail-fast to typed exceptions usage ended up being 10:1. Exceptions were generally used for I/O and things that dealt with user data, like the shell and parsers. Contracts were the biggest source of fail-fast.
The result was the best possible configuration of the above code quality attributes:
A nice accident of our model was that we could have compiled it with either return codes or exceptions. Thanks to this, we actually did the experiment, to see what the impact was to our system's size and speed. The exceptions-based system ended up being roughly 7% smaller and 4% faster on some key benchmarks.
At the end, what we ended up with was the most robust error model I've ever used, and certainly the most performant one.
As implied above, Midori's programming language had first class contracts:
void Push(T element)
requires element != null
ensures this.Count == old.Count + 1
{
...
}
The model was simple:
We had conditional compilation modes, however I will skip these for now. Look for an upcoming post on our language.
In the early days, we experimented with contract analyzers like MSR's Clousot, to prove contracts. For compile-time reasons, however, we had to abandon this approach. It turns out compilers are already very good at doing simple constraint solving and propagation. So eventually we just modeled contracts as facts that the compiler knew about, and let it insert the checks wherever necessary.
For example, the loop optimizer complete with range information above can already leverage checks like this:
void M(int[] array, int index) {
if (index >= 0 && index < array.Length) {
int v = array[index];
...
}
}
to eliminate the redundant bounds check inside the guarded if statement. So why not also do the same thing here?
void M(int[] array, int index)
requires index >= 0 && index < array.Length {
int v = array[index];
...
}
These facts were special, however, when it comes to separate compilation. A contract is part of a method's signature, and our system ensured proper subtyping substitution, letting the compiler do more aggressive optimizations at separately compiled boundaries. And it could do these optimizations faster because they didn't depend on global analysis.
In a future post, I'll describe in great detail our war with the garbage collector. One technique that helped us win, however, was to aggressively reduce the size and quantity of objects a well-behaving program allocated on the heap. This helped with overall working set and hence made programs smaller and faster.
The first technique here was to shrink object sizes.
In C# and most Java VMs, objects have headers. A standard size is a single word, that is, 4 bytes on 32-bit architectures and 8 bytes on 64-bit. This is in addition to the vtable pointer. It's typically used by the GC to mark objects and, in .NET, is used for random stuff, like COM interop, locking, memozation of hash codes, and more. (Even the source code calls it the "kitchen sink".)
Well, we ditched both.
We didn't have COM interop. There was no unsafe free-threading so there was no locking (and locking on random objects is a bad idea anyway). Our Object didn't define a GetHashCode. Etc. This saved a word per object with no discernable loss in the programming model (actually, to the contrary, it was improved), which is nothing to shake a stick at.
At that point, the only overhead per object was the vtable pointer. For structs, of course there wasn't one (unless they were boxed). And we did our best to eliminate all of them. Sadly, due to RTTI, it was difficult to be aggressive. I think this is another area where I'd go back and entirely upend the C# type system, to follow a more C, C++, or even maybe Go-like, model. In the end, however, I think we did get to be fairly competetive with your average C++ program.
There were padding challenges. Switching the struct layout from C#'s current default of sequential, to our preferred default of auto, certainly helped. As did optimizations like the well-known C++ empty base optimization.
We also did aggressive escape analysis in order to more efficiently allocate objects. If an object was found to be stack-confined, it was allocated on the stack instead of the heap. Our initial implementation of this moved somewhere in the neighborhood of 10% static allocations from the heap to the stack, and let us be far more aggressive about pruning back the size of objects, eliminating vtable pointers and entire unused fields. Given how conservative this analysis had to be, I was pretty happy with these results.
We offered a hybrid between C++ references and Rust borrowing if developers wanted to give the compiler a hint while at the same time semantically enforcing some level of containment. For example, say I wanted to allocate a little array to share with a callee, but know for sure the callee does not remember a reference to it. This was as simple as saying:
void Caller() {
Callee(new[] { 0, 1, ..., 9 });
}
void Callee(int[]& a) {
... guaranteed that `a` does not escape ...
}
The compiler used the int[]& information to stack allocate the array and, often, eliminating the vtable for it entirely. Coupled with the sophisticated elimination of bounds checking, this gave us something far closer to C performance.
Lambdas/delegates in our system were also structs, so did not require heap allocation. The captured display frame was subject to all of the above, so frequently we could stack allocate them. As a result, the following code was heap allocation-free; in fact, thanks to some early optimizations, if the callee was inlined, it ran as though the actual lambda body was merely expanded as a sequence of instructions, with no call over head either!
void Caller() {
Callee(() => ... do something ... );
}
void Callee(Action& callback) {
callback();
}
In my opinion, this really was the killer use case for the borrowing system. Developers avoided lambda-based APIs in the early days before we had this feature for fear of allocations and inefficiency. After doing this feature, on the other hand, a vibrant ecosystem of expressive lambda-based APIs flourished.
All of the above have to do with code quality; that is, the size and speed of the resulting code. Another important dimension of compiler performance, however, is throughput; that is, how quickly you can compile the code. Here too a language like C# comes with some of its own challenges.
The biggest challenge we encountered has less to do with the inherently safe nature of a language, and more to do with one very powerful feature: parametric polymorphism. Or, said less pretentiously, generics.
I already mentioned earlier that generics are just a convenient copy-and-paste mechanism. And I mentioned some challenges this poses for code size. It also poses a problem for throughput, however. If a List<T> instantiation creates 28 types, each with its own handful of methods, that's just more code for the compiler to deal with. Separate compilation helps, however as also noted earlier, generics often flow across module boundaries. As a result, there's likely to be a non-trivial impact to compile time. Indeed, there was.
In fact, this is not very different from where most C++ compilers spend the bulk of their time. In C++, it's templates. More modern C++ code-bases have similar problems, due to heavy use of templated abstractions, like STL, smart pointers, and the like. Many C++ code-bases are still just "C with classes" and suffer this problem less.
As I mentioned earlier, I wish we had banished RTTI. That would have lessened the generics problem. But I would guess generics still would have remained our biggest throughput challenge at the end of the day.
The funny thing -- in a not-so-funny kind of way -- is that you can try to do analysis to prune the set of generics and, though it is effective, this analysis takes time. The very thing you're trying to save.
A metric we got in the habit of tracking was how much slower AOT compiling a program was than simply C# compiling it. This was a totally unfair comparison, because the C# compiler just needs to lower to MSIL whereas an AOT compler needs to produce machine code. It'd have been fairer to compare AOT compiling to JIT compiling. But no matter, doing a great job on throughput is especially important for a C# audience. The expectation of productivity was quite high. This was therefore the key metric we felt customers would judge us on, and so we laser-focused on it.
In the early days, the number was ridiculously bad. I remember it being 40x slower. After about a year and half with intense focus we got it down to 3x for debug builds and 5x for optimized builds. I was very happy with this!
There was no one secret to achieving this. Mostly it had to do with just making the compiler faster like you would any program. Since we built the compiler using Midori's toolchain, however -- and compiled it using itself -- often this was done by first making Midori better, which then made the the compiler faster. It was a nice virtuous loop. We had real problems with string allocations which informed what to do with strings in our programming model. We found crazy generics instantiation closures which forced us to eliminate them and build tools to help find them proactively. Etc.
A final word before wrapping up. Culture was the most important aspect of what we did. Without the culture, such an amazing team wouldn't have self-selected, and wouldn't have relentlessly pursued all of the above achievements. I'll devote an entire post to this. However, in the context of compilers, two things helped:
Every sprint, we had a so-called "CQ Review" (where CQ stands for "code quality"). The compiler team prepared for a few days, by reviewing every benchmark -- ranging from the lowest of microbenchmarks to compiling and booting all of Windows -- and investigating any changes. All expected wins were confirmed (we called this "confirming your kill"), any unexpected regressions were root cause analyzed (and bugs filed), and any wins that didn't materialize were also analyzed and reported on so that we could learn from it. We even stared at numbers that didn't change, and asked ourselves, why didn't they change. Was it expected? Do we feel bad about it and, if so, how will we change next sprint? We reviewed our competitors' latest compiler drops and monitored their rate of change. And so on.
This process was enormously healthy. Everyone was encouraged to be self-critical. This was not a "witch hunt"; it was an opportunity to learn as a team how to do better at achieving our goals.
Post-Midori, I have kept this process. I've been surprised at how contentious this can be with some folks. They get threatened and worry their lack of progress makes them look bad. They use "the numbers aren't changing because that's not our focus right now" as justification for getting out of the rhythm. In my experience, so long as the code is changing, the numbers are changing. It's best to keep your eye on them lest you get caught with your pants around your ankles many months later when it suddenly matters most. The discipline and constant drumbeat are the most important parts of these reviews, so skipping even just one can be detrimental, and hence was verboten.
This process was as much our secret sauce as anything else was.
Whew, that was a lot of ground to cover. I hope at the very least it was interesting, and I hope for the incredible team who built all of this that I did it at least a fraction of justice. (I know I didn't.)
This journey took us over a decade, particularly if you account for the fact that both Bartok and Phoenix had existed for many years even before Midori formed. Merely AOT compiling C#, and doing it well, would have netted us many of the benefits above. But to truly achieve the magical native-like performance, and indeed even exceed it in certain areas, required some key "whole system" architectural bets. I hope that some day we can deliver safety into the world at this level of performance. Given the state of security all-up in the industry, mankind seriously needs it.
I've now touched on our programming language enough that I need to go deep on it. Tune in next time!
Physicists typically think they “need philosophers and historians of science like birds need ornithologists,” the Nobel laureate David Gross told a roomful of philosophers, historians and physicists last week in Munich, Germany, paraphrasing Richard Feynman.
But desperate times call for desperate measures.
Fundamental physics faces a problem, Gross explained — one dire enough to call for outsiders’ perspectives. “I’m not sure that we don’t need each other at this point in time,” he said.
It was the opening session of a three-day workshop, held in a Romanesque-style lecture hall at Ludwig Maximilian University (LMU Munich) one year after George Ellis and Joe Silk, two white-haired physicists now sitting in the front row, called for such a conference in an incendiary opinion piece in Nature. One hundred attendees had descended on a land with a celebrated tradition in both physics and the philosophy of science to wage what Ellis and Silk declared a “battle for the heart and soul of physics.”
The crisis, as Ellis and Silk tell it, is the wildly speculative nature of modern physics theories, which they say reflects a dangerous departure from the scientific method. Many of today’s theorists — chief among them the proponents of string theory and the multiverse hypothesis — appear convinced of their ideas on the grounds that they are beautiful or logically compelling, despite the impossibility of testing them. Ellis and Silk accused these theorists of “moving the goalposts” of science and blurring the line between physics and pseudoscience. “The imprimatur of science should be awarded only to a theory that is testable,” Ellis and Silk wrote, thereby disqualifying most of the leading theories of the past 40 years. “Only then can we defend science from attack.”
They were reacting, in part, to the controversial ideas of Richard Dawid, an Austrian philosopher whose 2013 book String Theory and the Scientific Method identified three kinds of “non-empirical” evidence that Dawid says can help build trust in scientific theories absent empirical data. Dawid, a researcher at LMU Munich, answered Ellis and Silk’s battle cry and assembled far-flung scholars anchoring all sides of the argument for the high-profile event last week.
Laetitia Vancon for Quanta Magazine
David Gross, a theoretical physicist at the University of California, Santa Barbara.
Gross, a supporter of string theory who won the 2004 Nobel Prize in physics for his work on the force that glues atoms together, kicked off the workshop by asserting that the problem lies not with physicists but with a “fact of nature” — one that we have been approaching inevitably for four centuries.
The dogged pursuit of a fundamental theory governing all forces of nature requires physicists to inspect the universe more and more closely — to examine, for instance, the atoms within matter, the protons and neutrons within those atoms, and the quarks within those protons and neutrons. But this zooming in demands evermore energy, and the difficulty and cost of building new machines increases exponentially relative to the energy requirement, Gross said. “It hasn’t been a problem so much for the last 400 years, where we’ve gone from centimeters to millionths of a millionth of a millionth of a centimeter” — the current resolving power of the Large Hadron Collider (LHC) in Switzerland, he said. “We’ve gone very far, but this energy-squared is killing us.”
As we approach the practical limits of our ability to probe nature’s underlying principles, the minds of theorists have wandered far beyond the tiniest observable distances and highest possible energies. Strong clues indicate that the truly fundamental constituents of the universe lie at a distance scale 10 million billion times smaller than the resolving power of the LHC. This is the domain of nature that string theory, a candidate “theory of everything,” attempts to describe. But it’s a domain that no one has the faintest idea how to access.
The problem also hampers physicists’ quest to understand the universe on a cosmic scale: No telescope will ever manage to peer past our universe’s cosmic horizon and glimpse the other universes posited by the multiverse hypothesis. Yet modern theories of cosmology lead logically to the possibility that our universe is just one of many.
Tynan DeBold for Quanta Magazine; Icons via Freepik
Whether the fault lies with theorists for getting carried away, or with nature, for burying its best secrets, the conclusion is the same: Theory has detached itself from experiment. The objects of theoretical speculation are now too far away, too small, too energetic or too far in the past to reach or rule out with our earthly instruments. So, what is to be done? As Ellis and Silk wrote, “Physicists, philosophers and other scientists should hammer out a new narrative for the scientific method that can deal with the scope of modern physics.”
“The issue in confronting the next step,” said Gross, “is not one of ideology but strategy: What is the most useful way of doing science?”
Over three mild winter days, scholars grappled with the meaning of theory, confirmation and truth; how science works; and whether, in this day and age, philosophy should guide research in physics or the other way around. Over the course of these pressing yet timeless discussions, a degree of consensus took shape.
Rules of the Game
Throughout history, the rules of science have been written on the fly, only to be revised to fit evolving circumstances. The ancients believed they could reason their way toward scientific truth. Then, in the 17th century, Isaac Newton ignited modern science by breaking with this “rationalist” philosophy, adopting instead the “empiricist” view that scientific knowledge derives only from empirical observation. In other words, a theory must be proved experimentally to enter the book of knowledge.
But what requirements must an untested theory meet to be considered scientific? Theorists guide the scientific enterprise by dreaming up the ideas to be put to the test and then interpreting the experimental results; what keeps theorists within the bounds of science?
Today, most physicists judge the soundness of a theory by using the Austrian-British philosopher Karl Popper’s rule of thumb. In the 1930s, Popper drew a line between science and nonscience in comparing the work of Albert Einstein with that of Sigmund Freud. Einstein’s theory of general relativity, which cast the force of gravity as curves in space and time, made risky predictions — ones that, if they hadn’t succeeded so brilliantly, would have failed miserably, falsifying the theory. But Freudian psychoanalysis was slippery: Any fault of your mother’s could be worked into your diagnosis. The theory wasn’t falsifiable, and so, Popper decided, it wasn’t science.
Laetitia Vancon for Quanta Magazine
Paul Teller (by window), a philosopher and professor emeritus at the University of California, Davis.
Critics accuse string theory and the multiverse hypothesis, as well as cosmic inflation— the leading theory of how the universe began — of falling on the wrong side of Popper’s line of demarcation. To borrow the title of the Columbia University physicist Peter Woit’s 2006 book on string theory, these ideas are “not even wrong,” say critics. In their editorial, Ellis and Silk invoked the spirit of Popper: “A theory must be falsifiable to be scientific.”
But, as many in Munich were surprised to learn, falsificationism is no longer the reigning philosophy of science. Massimo Pigliucci, a philosopher at the Graduate Center of the City University of New York, pointed out that falsifiability is woefully inadequate as a separator of science and nonscience, as Popper himself recognized. Astrology, for instance, is falsifiable — indeed, it has been falsified ad nauseam— and yet it isn’t science. Physicists’ preoccupation with Popper “is really something that needs to stop,” Pigliucci said. “We need to talk about current philosophy of science. We don’t talk about something that was current 50 years ago.”
Nowadays, as several philosophers at the workshop said, Popperian falsificationism has been supplanted by Bayesian confirmation theory, or Bayesianism, a modern framework based on the 18th-century probability theory of the English statistician and minister Thomas Bayes. Bayesianism allows for the fact that modern scientific theories typically make claims far beyond what can be directly observed — no one has ever seen an atom — and so today’s theories often resist a falsified-unfalsified dichotomy. Instead, trust in a theory often falls somewhere along a continuum, sliding up or down between 0 and 100 percent as new information becomes available. “The Bayesian framework is much more flexible” than Popper’s theory, said Stephan Hartmann, a Bayesian philosopher at LMU. “It also connects nicely to the psychology of reasoning.”
Gross concurred, saying that, upon learning about Bayesian confirmation theory from Dawid’s book, he felt “somewhat like the Molière character who said, ‘Oh my God, I’ve been talking prose all my life!’”
Another advantage of Bayesianism, Hartmann said, is that it is enabling philosophers like Dawid to figure out “how this non-empirical evidence fits in, or can be fit in.”
Another Kind of Evidence
Dawid, who is 49, mild-mannered and smiley with floppy brown hair, started his career as a theoretical physicist. In the late 1990s, during a stint at the University of California, Berkeley, a hub of string-theory research, Dawid became fascinated by how confident many string theorists seemed to be that they were on the right track, despite string theory’s complete lack of empirical support. “Why do they trust the theory?” he recalls wondering. “Do they have different ways of thinking about it than the canonical understanding?”
String theory says that elementary particles have dimensionality when viewed close-up, appearing as wiggling loops (or “strings”) and membranes at nature’s highest zoom level. According to the theory, extra dimensions also materialize in the fabric of space itself. The different vibrational modes of the strings in this higher-dimensional space give rise to the spectrum of particles that make up the observable world. In particular, one of the vibrational modes fits the profile of the “graviton” — the hypothetical particle associated with the force of gravity. Thus, string theory unifies gravity, now described by Einstein’s theory of general relativity, with the rest of particle physics.
Laetitia Vancon for Quanta Magazine
Video: Richard Dawid, a physicist-turned-philosopher at Ludwig Maximilian University in Munich.
However string theory, which has its roots in ideas developed in the late 1960s, has made no testable predictions about the observable universe. To understand why so many researchers trust it anyway, Dawid signed up for some classes in philosophy of science, and upon discovering how little study had been devoted to the phenomenon, he switched fields.
In the early 2000s, he identified three non-empirical arguments that generate trust in string theory among its proponents. First, there appears to be only one version of string theory capable of achieving unification in a consistent way (though it has many different mathematical representations); furthermore, no other “theory of everything” capable of unifying all the fundamental forces has been found, despite immense effort. (A rival approach called loop quantum gravity describes gravity at the quantum scale, but makes no attempt to unify it with the other forces.) This “no-alternatives” argument, colloquially known as “string theory is the only game in town,” boosts theorists’ confidence that few or no other possible unifications of the four fundamental forces exist, making it more likely that string theory is the right approach.
Second, string theory grew out of the Standard Model — the accepted, empirically validated theory incorporating all known fundamental particles and forces (apart from gravity) in a single mathematical structure — and the Standard Model also had no alternatives during its formative years. This “meta-inductive” argument, as Dawid calls it, buttresses the no-alternatives argument by showing that it has worked before in similar contexts, countering the possibility that physicists simply aren’t clever enough to find the alternatives that exist.
Emily Fuhrman for Quanta Magazine, with text by Natalie Wolchover and art direction by Olena Shmahalo.
The third non-empirical argument is that string theory has unexpectedly delivered explanations for several other theoretical problems aside from the unification problem it was intended to address. The staunch string theorist Joe Polchinski of the University of California, Santa Barbara, presented several examples of these “unexpected explanatory interconnections,” as Dawid has termed them, in a paper read in Munich in his absence. String theory explains the entropy of black holes, for example, and, in a surprising discovery that has caused a surge of research in the past 15 years, is mathematically translatable into a theory of particles, such as the theory describing the nuclei of atoms.
Polchinski concludes that, considering how far away we are from the exceptionally fine grain of nature’s fundamental distance scale, we should count ourselves lucky: “String theory exists, and we have found it.” (Polchinski also used Dawid’s non-empirical arguments to calculate the Bayesian odds that the multiverse exists as 94 percent — a value that has been ridiculed by the Internet’s vocal multiverse critics.)
One concern with including non-empirical arguments in Bayesian confirmation theory, Dawid acknowledged in his talk, is “that it opens the floodgates to abandoning all scientific principles.” One can come up with all kinds of non-empirical virtues when arguing in favor of a pet idea. “Clearly the risk is there, and clearly one has to be careful about this kind of reasoning,” Dawid said. “But acknowledging that non-empirical confirmation is part of science, and has been part of science for quite some time, provides a better basis for having that discussion than pretending that it wasn’t there, and only implicitly using it, and then saying I haven’t done it. Once it’s out in the open, one can discuss the pros and cons of those arguments within a specific context.”
The Munich Debate
Laetitia Vancon for Quanta Magazine
Except sometimes it is. Rationalism guided Einstein toward his theory of relativity, which he believed in wholeheartedly on rational grounds before it was ever tested. “I hold it true that pure thought can grasp reality, as the ancients dreamed,” Einstein said in 1933, years after his theory had been confirmed by observations of starlight bending around the sun.
The question for the philosophers is: Without experiments, is there any way to distinguish between the non-empirical virtues of vortex theory and those of Einstein’s theory? Can we ever really trust a theory on non-empirical grounds?
In discussions on the third afternoon of the workshop, the LMU philosopher Radin Dardashti asserted that Dawid’s philosophy specifically aims to pinpoint which non-empirical arguments should carry weight, allowing scientists to “make an assessment that is not based on simplicity, which is not based on beauty.” Dawidian assessment is meant to be more objective than these measures, Dardashti explained — and more revealing of a theory’s true promise.
Gross said Dawid has “described beautifully” the strategies physicists use “to gain confidence in a speculation, a new idea, a new theory.”
“You mean confidence that it’s true?” asked Peter Achinstein, an 80-year-old philosopher and historian of science at Johns Hopkins University. “Confidence that it’s useful? confidence that …”
“Let’s give an operational definition of confidence: I will continue to work on it,” Gross said.
“That’s pretty low,” Achinstein said.
“Not for science,” Gross said. “That’s the question that matters.”
Kragh pointed out that even Popper saw value in the kind of thinking that motivates string theorists today. Popper called speculation that did not yield testable predictions “metaphysics,” but he considered such activity worthwhile, since it might become testable in the future. This was true of atomic theory, which many 19th-century physicists feared would never be empirically confirmed. “Popper was not a naive Popperian,” Kragh said. “If a theory is not falsifiable,” Kragh said, channeling Popper, “it should not be given up. We have to wait.”
But several workshop participants raised qualms about Bayesian confirmation theory, and about Dawid’s non-empirical arguments in particular.
Related Articles:
Visions of Future Physics
Nima Arkani-Hamed is championing a campaign to build the world’s largest particle collider, even as he pursues a new vision of the laws of nature.In a Multiverse, What Are the Odds?
Testing the multiverse hypothesis requires measuring whether our universe is statistically typical among the infinite variety of universes. But infinity does a number on statistics.In Fake Universes, Evidence for String Theory
Researchers are demonstrating that, in certain contexts, string theory is the only consistent theory of quantum gravity. Might this make it true?
Carlo Rovelli, a proponent of loop quantum gravity (string theory’s rival) who is based at Aix-Marseille University in France, objected that Bayesian confirmation theory does not allow for an important distinction that exists in science between theories that scientists are certain about and those that are still being tested. The Bayesian “confirmation” that atoms exist is essentially 100 percent, as a result of countless experiments. But Rovelli says that the degree of confirmation of atomic theory shouldn’t even be measured in the same units as that of string theory. String theory is not, say, 10 percent as confirmed as atomic theory; the two have different statuses entirely. “The problem with Dawid’s ‘non-empirical confirmation’ is that it muddles the point,” Rovelli said. “And of course some string theorists are happy of muddling it this way, because they can then say that string theory is ‘confirmed,’ equivocating.”
The German physicist Sabine Hossenfelder, in her talk, argued that progress in fundamental physics very often comes from abandoning cherished prejudices (such as, perhaps, the assumption that the forces of nature must be unified). Echoing this point, Rovelli said “Dawid’s idea of non-empirical confirmation [forms] an obstacle to this possibility of progress, because it bases our credence on our own previous credences.” It “takes away one of the tools — maybe the soul itself — of scientific thinking,” he continued, “which is ‘do not trust your own thinking.’”
The Munich proceedings will be compiled and published, probably as a book, in 2017. As for what was accomplished, one important outcome, according to Ellis, was an acknowledgment by participating string theorists that the theory is not “confirmed” in the sense of being verified. “David Gross made his position clear: Dawid’s criteria are good for justifying working on the theory, not for saying the theory is validated in a non-empirical way,” Ellis wrote in an email. “That seems to me a good position — and explicitly stating that is progress.”
In considering how theorists should proceed, many attendees expressed the view that work on string theory and other as-yet-untestable ideas should continue. “Keep speculating,” Achinstein wrote in an email after the workshop, but “give your motivation for speculating, give your explanations, but admit that they are only possible explanations.”
“Maybe someday things will change,” Achinstein added, “and the speculations will become testable; and maybe not, maybe never.” We may never know for sure the way the universe works at all distances and all times, “but perhaps you can narrow the live possibilities to just a few,” he said. “I think that would be some progress.”
An Uber-style business that connects private pilots with travelers willing to split fuel costs and other expenses was shot down by a Washington, D.C., court.
The judges on Friday declined Flytenow Inc.’s request to review a Federal Aviation Administration ruling that pilots who use the service to find passengers must have commercial licenses.
Flytenow.com connects members who share expenses in exchange for flights on a route predetermined by the pilot. AirPooler Inc. offers a similar service that was also blocked by the FAA’s rules, which rejected the idea that cost-sharing is different from a commercial aviation operation.
The company asked the court to consider whether the FAA was permitted to conclude that private pilots using its service violated their non-commercial licenses by representing themselves as common carriers for compensation. The court said yes.
The FAA is justified in making a distinction between pilots offering expense-sharing services to a limited group and those offering the same services to a wide audience, the court said. There’s a risk that unsuspecting passengers “will contract with pilots who in fact lack the experience and credentials of commercial pilots,” the court said in its ruling.
While private pilots must adhere to FAA safety regulations, they receive significantly less scrutiny.
“Regulators have good reasons to distinguish between pilots who are licensed to offer services to the public and those who are not, as other courts have recognized,” a three-judge panel ruled.
The case is Flytenow Inc. v. Federal Aviation Administration, 14-1168, U.S. Court of Appeals for the District of Columbia Circuit (Washington).
Related ticker:
0084207D (Uber Technologies)
Anti Debugging Tricks By: Inbar Raz Assistance by Eden Shochat and Yossi Gottlieb Release number 5 Today's anti debugging tricks devide into two categories: 1. Preventive actions; 2. Self-modifying code. Most debugging tricks, as for today, are used within viruses, in order to avoid dis-assembly of the virus, as it will be exampled later in this file. Another large portion of anti debugging tricks is found with software protection programs, that use them in order to make the cracking of the protection harder. 1. Preventive actions: ---------------------- Preventive actions are, basically, actions that the program takes in order to make the user unable to dis-assemble the code or trace it while running. 1.1. Interrupt disable: Interrupt disable is probably the most common form of anti-debugging tricks. It can be done in several ways: 1.1.1. Hardware masking of interrupt: In order to avoid tracing of a code, one usually disables the interrupt via the 8259 Interrupt Controller, addressed by read/write actions to port 21h. The 8259 Interrupt Controller controls the IRQ lines. This means that any IRQ between 0 and 7 may be disabled by this action. Bit 0 is IRQ0, bit 1 is IRQ1 etc. Since IRQ1 is the keyboard interrupt, you may disable the keyboard without the debugger being able to bypass it. Example: CS:0100 E421 IN AL,21 CS:0102 0C02 OR AL,02 CS:0104 E621 OUT 21,AL Just as a side notice, the keyboard may be also disabled by commanding the Programmable Peripheral Interface (PPI), port 61h. Example: CS:0100 E461 IN AL,61 CS:0102 0C80 OR AL,80 CS:0104 E661 OUT 61,AL 1.1.2. Software masking of interrupt: This is quite an easy form of an anti-debugging trick. All you have to do is simply replace the vectors of interrupts debuggers use, or any other interrupt you will not be using or expecting to occur. Do not forget to restore the original vectors when you are finished. It is adviseable to use manual change of vector, as shown below, rather than to change it using interrupt 21h service 25h, because any debugger that has gained control of interrupt 21h may replace your vector with the debugger's. The example shows an interception of interrupt 03h - the breakpoint interrupt. Example: CS:0100 EB04 JMP 0106 CS:0102 0000 ADD [BX+SI],AL CS:0104 0000 ADD [BX+SI],AL CS:0106 31C0 XOR AX,AX CS:0108 8EC0 MOV ES,AX CS:010A 268B1E0C00 MOV BX,ES:[000C] CS:010F 891E0201 MOV [0102],BX CS:0113 268B1E0E00 MOV BX,ES:[000E] CS:0118 891E0401 MOV [0104],BX CS:011C 26C7064C000000 MOV Word Ptr ES:[000C],0000 CS:0123 26C7064E000000 MOV Word Ptr ES:[000E],0000 1.1.3. Vector manipulation This method involves manipulations of the interrupt vectors, mainly for proper activation of the algorithm. Such action, as exampled, may be used to decrypt a code (see also 2.1), using data stored ON the vectors. Ofcourse, during normal operation of the program, vectors 01h and 03h are not used, so unless you are trying to debug such a program, it works fine. Example: CS:0100 31C0 XOR AX,AX CS:0102 8ED0 MOV SS,AX CS:0104 BC0E00 MOV SP,000E CS:0107 2E8B0E3412 MOV CX,CS:[1234] CS:010C 50 PUSH AX CS:010D 31C8 XOR AX,CX CS:010F 21C5 AND BP,AX CS:0111 58 POP AX CS:0112 E2F8 LOOP 010C 1.1.4. Interrupt replacement This is a really nasty trick, and it should be used ONLY if you are ABSOLUTELY sure that your programs needs no more debugging. What you should do is copy the vectors of some interrupts you will be using, say 16h and 21h, onto the vectors of interrupt 01h and 03h, that do not occur during normal operation of the program. If the user wants to debug the program, he would have to search for every occurance of INT 01, and replace it with the appropriate INT instruction. This trick is very effective if used together with the fact that the INT 3 intruction has a ONE BYTE opcode - 0CCh, which can not be changed to any other interrupt. Example: CS:0100 FA CLI CS:0101 31C0 XOR AX,AX CS:0103 8EC0 MOV ES,AX CS:0105 26A18400 MOV AX,ES:[0084] CS:0109 26A30400 MOV ES:[0004],AX CS:010D 26A18600 MOV AX,ES:[0086] CS:0111 26A30600 MOV ES:[0006],AX CS:0115 B44C MOV AH,4C CS:0117 CD01 INT 01 1.2. Time watch: This may be a less common method, but it is usefull against debuggers that disable all interrupts except for the time that the program is executed, such as Borland's Turbo Debugger. This method simply retains the value of the clock counter, updated by interrupt 08h, and waits in an infinite loop until the value changes. Another example is when you mask the timer interrupt by ORing the value INed from port 21h with 01h and then OUTing it back, thus disabling the IRQ0 - Timer interrupt. Note that this method is usefull only against RUN actions, not TRACE/PROCEED ones. Example: CS:0100 2BC0 SUB AX,AX CS:0102 FB STI CS:0103 8ED8 MOV DS,AX CS:0105 8A266C04 MOV AH,[046C] CS:0109 A06C04 MOV AL,[046C] CS:010C 3AC4 CMP AL,AH CS:010E 74F9 JZ 0109 1.3. Fool the debugger: This is a very nice technique, that works especially and only on those who use Turbo Debugger or its kind. What you should do is init a jump to a middle of an instruction, whereas the real address actually contains another opcode. If you work with a normal step debugger such as Debug or SymDeb, it won't work since the debugger jumps to the exact address of the jump, and not to the beginning of an instruction at the closest address, like Turbo Debugger. Example: CS:0100 E421 IN AL,21 CS:0102 B0FF MOV AL,FF CS:0104 EB02 JMP 0108 CS:0106 C606E62100 MOV Byte Ptr [21E6],00 CS:010B CD20 INT 20 Watch this: CS:0108 E621 OUT 21,AL Notice: This trick does NOT effect the run of the program in ANY debugger. Its only use is to try to deceive the user into thinking another opcode is used, while another is actually run. 1.4. Check CPU Flags: This is a nice trick, effective against almost any real mode debugger. What you should do is simply set the trace flag off somewhere in your program, and check for it later. If it was turned on, a debugger runs in the background... Example: CS:0100 9C PUSHF CS:0101 58 POP AX CS:0102 25FFFE AND AX,FEFF CS:0105 50 PUSH AX CS:0106 9D POPF In the middle of the program: CS:1523 9C PUSHF CS:1524 58 POP AX CS:1525 250001 AND AX,0100 CS:1528 7402 JZ 152C CS:152A CD20 INT 20 1.5. Cause debugger to stop execution: This is a technique that causes a debugger to stop the execution of a certain program. What you need to do is to put some INT 3 instructions over the code, at random places, and any debugger trying to run will stop there. It is best if used within a loop, as it is run several times. Example: CS:0100 B96402 MOV CX,0264 CS:0103 BE1001 MOV SI,0110 CS:0106 AC LODSB CS:0107 CC INT 3 CS:0108 98 CBW CS:0109 01C3 ADD BX,AX CS:010B E2F9 LOOP 0106 1.6. Halt computer using stack: This trick is based on the fact that debuggers don't usually use a stack space of their own, but rather the user program's stack space. By setting the stack to a location in the middle of a code that does NOT use the stack itself, any debugger that will try to trace the code will overwrite some of the code by its own stack (mainly interrupt return addresses). Again, CLI and STI are in order, and are not shown for the purpose of the example only. They must be included, or you risk hanging your computer wether a debugger is installed or not. Example: CS:0100 8CD0 MOV AX,SS CS:0102 89E3 MOV BX,SP CS:0104 0E PUSH CS CS:0105 17 POP SS CS:0106 BC0B01 MOV SP,010B CS:0109 90 NOP CS:010A 90 NOP CS:010B EB02 JMP 010F CS:010D 90 NOP CS:010E 90 NOP CS:010F 89DC MOV SP,BX CS:0111 8ED0 MOV SS,AX 1.7. Halt TD386 V8086 mode: This is a nice way to fool Turbo Debugger's V8086 module (TD386). It is based on the fact that TD386 does not use INT 00h to detect division by zero (or register overrun after division, which is treated by the processor in the same way as in the case of division by zero). When TD386 detects a division fault, it aborts, reporting about the faulty division. In real mode (even under a regular debugger), a faulty DIV instruction will cause INT 00h to be called. Therefore, pointing INT 00h to the next instruction, will recover from the faulty DIV. Note: It is very important to restore INT 00h's vector. Otherwise, the next call to INT 00h will cause the machine to hang. Example: CS:0100 31C0 XOR AX,AX CS:0102 8ED8 MOV DS,AX CS:0104 C70600001201 MOV WORD PTR [0000],0112 CS:010A 8C0E0200 MOV [0002],CS CS:010E B400 MOV AH,00 CS:0110 F6F4 DIV AH CS:0112 B8004C MOV AX,4C00 CS:0115 CD21 INT 21 1.8. Halt any V8086 process: Another way of messing TD386 is fooling it into an exception. Unfortunately, this exception will also be generated under any other program, running at V8086 mode. The exception is exception #13, and its issued interrupt is INT 0Dh - 13d. The idea is very similar to the divide by zero trick: Causing an exception, when the exception interrupt points to somewhere in the program's code. It will always work when the machine is running in real mode, but never under the V8086 mode. Note: It is very important to restore the original interrupt vectors. Otherwise, the next exception will hang the machine. Example: CS:0100 31C0 XOR AX,AX CS:0102 8ED8 MOV DS,AX CS:0104 C70634001301 MOV WORD PTR [0034],0113 CS:010A 8C0E3600 MOV [0036],CS CS:010E 833EFFFF00 CMP WORD PTR [FFFF],+00 CS:0113 B8004C MOV AX,4C00 CS:0116 CD21 INT 21 2. Self-modifying code: ----------------------- 2.1. Encryptive/decryptive algorithm: The first category is simply a code, that has been encrypted, and has been added a decryption routine. The trick here is that when a debugger sets up a breakpoint, it simply places the opcode CCh (INT 03h) in the desired address, and once that interrupt is executed, the debugger regains control of things. If you try to set a breakpoint AFTER the decryption algorithm, what is usually needed, you will end up putting an opcode CCh in a place where decryptive actions are taken, therefore losing your original CCh in favour of whatever the decryption algorithm produces. The following example was extracted from the Haifa virus. If you try to set a breakpoint at address CS:0110, you will never reach that address, since there is no way to know what will result from the change. Note that if you want to make the tracing even harder, you should start the decryption of the code from its END, so it takes the whole operation until the opcode following the decryption routine is decrypted. Example: CS:0100 BB7109 MOV BX,0971 CS:0103 BE1001 MOV DI,0110 CS:0106 91 XCHG AX,CX CS:0107 91 XCHG AX,CX CS:0108 2E803597 XOR Byte Ptr CS:[DI],97 CS:010C 47 INC DI CS:010D 4B DEC BX CS:010E 75F6 JNZ 0106 CS:0110 07 POP ES CS:0111 07 POP ES 2.2. Self-modifying code: 2.2.1. Simple self-modification: This method implements the same principle as the encryption method: Change the opcode before using it. In the following example, we change the insruction following the call, and therefore, if you try to trace the entire call ('P'/Debug or F8/Turbo Debugger), you will not succeed, since the debugger will put its CCh on offset 103h, but when the routine runs, it overwrites location 103h. Example: CS:0100 E80400 CALL 0107 CS:0103 CD20 INT 20 CS:0105 CD21 INT 21 CS:0107 C7060301B44C MOV Word Ptr [0103],4CB4 CS:010D C3 RET Watch this: CS:0103 B44C MOV AH,4C 2.2.2. The Running Line (self-decrypting): This is an example of a self-tracing self-modifying code, sometimes called 'The running line'. It was presented by Serge Pachkovsky. It is a bit tricky in implementation, but, unlike all other techiniques mentioned in this document, it is relatively resistive to various protections of the vector table. In short, it results in instructions being decoded one at time, thus never exposing long code fragments to analisys. I will illustrate it with the following (over-simplified) code example: XOR AX, AX MOV ES, AX MOV WORD PTR ES:[4*1+0],OFFSET TRACER MOV WORD PTR ES:[4*1+2],CS MOV BP, SP PUSHF XOR BYTE PTR [BP-1], 1 POPF MOV AX, 4C00H ; This will not be traced! DB 3 DUP ( 98H ) DB C5H, 21H TRACER: PUSH BP MOV BP, SP MOV BP, WORD PTR [BP+2] XOR BYTE PTR CS:[BP-1], 8 XOR BYTE PTR CS:[BP+0], 8 POP BP IRET =============================================================================== Comments: In order to save lines of code, I did not insert the CLI/STI pair before any vector change. However, it is adviseable to do this pair before ANY manual vector change, because if any interrupt occurs in the middle of your operations, the machine could hang. An apology: In previous releases of this article, a false example, as noted by Serge Pachkovksy, was posted. That was 2.2.2 - Manipulating the PIQ. Apperantly the posted source would not work under any circumstances. In return, Serge has presented the 'Running Line' technique. Thanks to: Eden Shochat, 2:401/100 and Yossi Gottlieb, 2:401/100.3 for helping me assembling this list. Other acknowledgements: Matt Pritchard, 80XXX echo Serge Pachkovsky, Distributed Node (2:5000/19.19) ================================================================================ Any comments, suggestions, ideas and corrections will be gladly accepted. Author can be reached in one of the following ways: Inbar Raz, 2:401/100.1 {fidonet} Inbar Raz, 2:403/100.42 {fidonet} nyvirus@weizmann.weizmann.ac.il {internet} uunet!m2xenix!puddle!2!403!100.42!Inbar.Raz {UUCP} Inbar.Raz@p1.f100.n401.z2.fidonet.org {internet<>FIDO gate} Inbar.Raz@p42.f100.n403.z2.fidonet.org {internet<>FIDO gate}
Anti Debugging Tricks By: Inbar Raz Assistance by Eden Shochat and Yossi Gottlieb Release number 5 Today's anti debugging tricks devide into two categories: 1. Preventive actions; 2. Self-modifying code. Most debugging tricks, as for today, are used within viruses, in order to avoid dis-assembly of the virus, as it will be exampled later in this file. Another large portion of anti debugging tricks is found with software protection programs, that use them in order to make the cracking of the protection harder. 1. Preventive actions: ---------------------- Preventive actions are, basically, actions that the program takes in order to make the user unable to dis-assemble the code or trace it while running. 1.1. Interrupt disable: Interrupt disable is probably the most common form of anti-debugging tricks. It can be done in several ways: 1.1.1. Hardware masking of interrupt: In order to avoid tracing of a code, one usually disables the interrupt via the 8259 Interrupt Controller, addressed by read/write actions to port 21h. The 8259 Interrupt Controller controls the IRQ lines. This means that any IRQ between 0 and 7 may be disabled by this action. Bit 0 is IRQ0, bit 1 is IRQ1 etc. Since IRQ1 is the keyboard interrupt, you may disable the keyboard without the debugger being able to bypass it. Example: CS:0100 E421 IN AL,21 CS:0102 0C02 OR AL,02 CS:0104 E621 OUT 21,AL Just as a side notice, the keyboard may be also disabled by commanding the Programmable Peripheral Interface (PPI), port 61h. Example: CS:0100 E461 IN AL,61 CS:0102 0C80 OR AL,80 CS:0104 E661 OUT 61,AL 1.1.2. Software masking of interrupt: This is quite an easy form of an anti-debugging trick. All you have to do is simply replace the vectors of interrupts debuggers use, or any other interrupt you will not be using or expecting to occur. Do not forget to restore the original vectors when you are finished. It is adviseable to use manual change of vector, as shown below, rather than to change it using interrupt 21h service 25h, because any debugger that has gained control of interrupt 21h may replace your vector with the debugger's. The example shows an interception of interrupt 03h - the breakpoint interrupt. Example: CS:0100 EB04 JMP 0106 CS:0102 0000 ADD [BX+SI],AL CS:0104 0000 ADD [BX+SI],AL CS:0106 31C0 XOR AX,AX CS:0108 8EC0 MOV ES,AX CS:010A 268B1E0C00 MOV BX,ES:[000C] CS:010F 891E0201 MOV [0102],BX CS:0113 268B1E0E00 MOV BX,ES:[000E] CS:0118 891E0401 MOV [0104],BX CS:011C 26C7064C000000 MOV Word Ptr ES:[000C],0000 CS:0123 26C7064E000000 MOV Word Ptr ES:[000E],0000 1.1.3. Vector manipulation This method involves manipulations of the interrupt vectors, mainly for proper activation of the algorithm. Such action, as exampled, may be used to decrypt a code (see also 2.1), using data stored ON the vectors. Ofcourse, during normal operation of the program, vectors 01h and 03h are not used, so unless you are trying to debug such a program, it works fine. Example: CS:0100 31C0 XOR AX,AX CS:0102 8ED0 MOV SS,AX CS:0104 BC0E00 MOV SP,000E CS:0107 2E8B0E3412 MOV CX,CS:[1234] CS:010C 50 PUSH AX CS:010D 31C8 XOR AX,CX CS:010F 21C5 AND BP,AX CS:0111 58 POP AX CS:0112 E2F8 LOOP 010C 1.1.4. Interrupt replacement This is a really nasty trick, and it should be used ONLY if you are ABSOLUTELY sure that your programs needs no more debugging. What you should do is copy the vectors of some interrupts you will be using, say 16h and 21h, onto the vectors of interrupt 01h and 03h, that do not occur during normal operation of the program. If the user wants to debug the program, he would have to search for every occurance of INT 01, and replace it with the appropriate INT instruction. This trick is very effective if used together with the fact that the INT 3 intruction has a ONE BYTE opcode - 0CCh, which can not be changed to any other interrupt. Example: CS:0100 FA CLI CS:0101 31C0 XOR AX,AX CS:0103 8EC0 MOV ES,AX CS:0105 26A18400 MOV AX,ES:[0084] CS:0109 26A30400 MOV ES:[0004],AX CS:010D 26A18600 MOV AX,ES:[0086] CS:0111 26A30600 MOV ES:[0006],AX CS:0115 B44C MOV AH,4C CS:0117 CD01 INT 01 1.2. Time watch: This may be a less common method, but it is usefull against debuggers that disable all interrupts except for the time that the program is executed, such as Borland's Turbo Debugger. This method simply retains the value of the clock counter, updated by interrupt 08h, and waits in an infinite loop until the value changes. Another example is when you mask the timer interrupt by ORing the value INed from port 21h with 01h and then OUTing it back, thus disabling the IRQ0 - Timer interrupt. Note that this method is usefull only against RUN actions, not TRACE/PROCEED ones. Example: CS:0100 2BC0 SUB AX,AX CS:0102 FB STI CS:0103 8ED8 MOV DS,AX CS:0105 8A266C04 MOV AH,[046C] CS:0109 A06C04 MOV AL,[046C] CS:010C 3AC4 CMP AL,AH CS:010E 74F9 JZ 0109 1.3. Fool the debugger: This is a very nice technique, that works especially and only on those who use Turbo Debugger or its kind. What you should do is init a jump to a middle of an instruction, whereas the real address actually contains another opcode. If you work with a normal step debugger such as Debug or SymDeb, it won't work since the debugger jumps to the exact address of the jump, and not to the beginning of an instruction at the closest address, like Turbo Debugger. Example: CS:0100 E421 IN AL,21 CS:0102 B0FF MOV AL,FF CS:0104 EB02 JMP 0108 CS:0106 C606E62100 MOV Byte Ptr [21E6],00 CS:010B CD20 INT 20 Watch this: CS:0108 E621 OUT 21,AL Notice: This trick does NOT effect the run of the program in ANY debugger. Its only use is to try to deceive the user into thinking another opcode is used, while another is actually run. 1.4. Check CPU Flags: This is a nice trick, effective against almost any real mode debugger. What you should do is simply set the trace flag off somewhere in your program, and check for it later. If it was turned on, a debugger runs in the background... Example: CS:0100 9C PUSHF CS:0101 58 POP AX CS:0102 25FFFE AND AX,FEFF CS:0105 50 PUSH AX CS:0106 9D POPF In the middle of the program: CS:1523 9C PUSHF CS:1524 58 POP AX CS:1525 250001 AND AX,0100 CS:1528 7402 JZ 152C CS:152A CD20 INT 20 1.5. Cause debugger to stop execution: This is a technique that causes a debugger to stop the execution of a certain program. What you need to do is to put some INT 3 instructions over the code, at random places, and any debugger trying to run will stop there. It is best if used within a loop, as it is run several times. Example: CS:0100 B96402 MOV CX,0264 CS:0103 BE1001 MOV SI,0110 CS:0106 AC LODSB CS:0107 CC INT 3 CS:0108 98 CBW CS:0109 01C3 ADD BX,AX CS:010B E2F9 LOOP 0106 1.6. Halt computer using stack: This trick is based on the fact that debuggers don't usually use a stack space of their own, but rather the user program's stack space. By setting the stack to a location in the middle of a code that does NOT use the stack itself, any debugger that will try to trace the code will overwrite some of the code by its own stack (mainly interrupt return addresses). Again, CLI and STI are in order, and are not shown for the purpose of the example only. They must be included, or you risk hanging your computer wether a debugger is installed or not. Example: CS:0100 8CD0 MOV AX,SS CS:0102 89E3 MOV BX,SP CS:0104 0E PUSH CS CS:0105 17 POP SS CS:0106 BC0B01 MOV SP,010B CS:0109 90 NOP CS:010A 90 NOP CS:010B EB02 JMP 010F CS:010D 90 NOP CS:010E 90 NOP CS:010F 89DC MOV SP,BX CS:0111 8ED0 MOV SS,AX 1.7. Halt TD386 V8086 mode: This is a nice way to fool Turbo Debugger's V8086 module (TD386). It is based on the fact that TD386 does not use INT 00h to detect division by zero (or register overrun after division, which is treated by the processor in the same way as in the case of division by zero). When TD386 detects a division fault, it aborts, reporting about the faulty division. In real mode (even under a regular debugger), a faulty DIV instruction will cause INT 00h to be called. Therefore, pointing INT 00h to the next instruction, will recover from the faulty DIV. Note: It is very important to restore INT 00h's vector. Otherwise, the next call to INT 00h will cause the machine to hang. Example: CS:0100 31C0 XOR AX,AX CS:0102 8ED8 MOV DS,AX CS:0104 C70600001201 MOV WORD PTR [0000],0112 CS:010A 8C0E0200 MOV [0002],CS CS:010E B400 MOV AH,00 CS:0110 F6F4 DIV AH CS:0112 B8004C MOV AX,4C00 CS:0115 CD21 INT 21 1.8. Halt any V8086 process: Another way of messing TD386 is fooling it into an exception. Unfortunately, this exception will also be generated under any other program, running at V8086 mode. The exception is exception #13, and its issued interrupt is INT 0Dh - 13d. The idea is very similar to the divide by zero trick: Causing an exception, when the exception interrupt points to somewhere in the program's code. It will always work when the machine is running in real mode, but never under the V8086 mode. Note: It is very important to restore the original interrupt vectors. Otherwise, the next exception will hang the machine. Example: CS:0100 31C0 XOR AX,AX CS:0102 8ED8 MOV DS,AX CS:0104 C70634001301 MOV WORD PTR [0034],0113 CS:010A 8C0E3600 MOV [0036],CS CS:010E 833EFFFF00 CMP WORD PTR [FFFF],+00 CS:0113 B8004C MOV AX,4C00 CS:0116 CD21 INT 21 2. Self-modifying code: ----------------------- 2.1. Encryptive/decryptive algorithm: The first category is simply a code, that has been encrypted, and has been added a decryption routine. The trick here is that when a debugger sets up a breakpoint, it simply places the opcode CCh (INT 03h) in the desired address, and once that interrupt is executed, the debugger regains control of things. If you try to set a breakpoint AFTER the decryption algorithm, what is usually needed, you will end up putting an opcode CCh in a place where decryptive actions are taken, therefore losing your original CCh in favour of whatever the decryption algorithm produces. The following example was extracted from the Haifa virus. If you try to set a breakpoint at address CS:0110, you will never reach that address, since there is no way to know what will result from the change. Note that if you want to make the tracing even harder, you should start the decryption of the code from its END, so it takes the whole operation until the opcode following the decryption routine is decrypted. Example: CS:0100 BB7109 MOV BX,0971 CS:0103 BE1001 MOV DI,0110 CS:0106 91 XCHG AX,CX CS:0107 91 XCHG AX,CX CS:0108 2E803597 XOR Byte Ptr CS:[DI],97 CS:010C 47 INC DI CS:010D 4B DEC BX CS:010E 75F6 JNZ 0106 CS:0110 07 POP ES CS:0111 07 POP ES 2.2. Self-modifying code: 2.2.1. Simple self-modification: This method implements the same principle as the encryption method: Change the opcode before using it. In the following example, we change the insruction following the call, and therefore, if you try to trace the entire call ('P'/Debug or F8/Turbo Debugger), you will not succeed, since the debugger will put its CCh on offset 103h, but when the routine runs, it overwrites location 103h. Example: CS:0100 E80400 CALL 0107 CS:0103 CD20 INT 20 CS:0105 CD21 INT 21 CS:0107 C7060301B44C MOV Word Ptr [0103],4CB4 CS:010D C3 RET Watch this: CS:0103 B44C MOV AH,4C 2.2.2. The Running Line (self-decrypting): This is an example of a self-tracing self-modifying code, sometimes called 'The running line'. It was presented by Serge Pachkovsky. It is a bit tricky in implementation, but, unlike all other techiniques mentioned in this document, it is relatively resistive to various protections of the vector table. In short, it results in instructions being decoded one at time, thus never exposing long code fragments to analisys. I will illustrate it with the following (over-simplified) code example: XOR AX, AX MOV ES, AX MOV WORD PTR ES:[4*1+0],OFFSET TRACER MOV WORD PTR ES:[4*1+2],CS MOV BP, SP PUSHF XOR BYTE PTR [BP-1], 1 POPF MOV AX, 4C00H ; This will not be traced! DB 3 DUP ( 98H ) DB C5H, 21H TRACER: PUSH BP MOV BP, SP MOV BP, WORD PTR [BP+2] XOR BYTE PTR CS:[BP-1], 8 XOR BYTE PTR CS:[BP+0], 8 POP BP IRET =============================================================================== Comments: In order to save lines of code, I did not insert the CLI/STI pair before any vector change. However, it is adviseable to do this pair before ANY manual vector change, because if any interrupt occurs in the middle of your operations, the machine could hang. An apology: In previous releases of this article, a false example, as noted by Serge Pachkovksy, was posted. That was 2.2.2 - Manipulating the PIQ. Apperantly the posted source would not work under any circumstances. In return, Serge has presented the 'Running Line' technique. Thanks to: Eden Shochat, 2:401/100 and Yossi Gottlieb, 2:401/100.3 for helping me assembling this list. Other acknowledgements: Matt Pritchard, 80XXX echo Serge Pachkovsky, Distributed Node (2:5000/19.19) ================================================================================ Any comments, suggestions, ideas and corrections will be gladly accepted. Author can be reached in one of the following ways: Inbar Raz, 2:401/100.1 {fidonet} Inbar Raz, 2:403/100.42 {fidonet} nyvirus@weizmann.weizmann.ac.il {internet} uunet!m2xenix!puddle!2!403!100.42!Inbar.Raz {UUCP} Inbar.Raz@p1.f100.n401.z2.fidonet.org {internet<>FIDO gate} Inbar.Raz@p42.f100.n403.z2.fidonet.org {internet<>FIDO gate}
To cite this book in preparation, please use the following bibtex entry:
![[No Caption]](http://www.quantamagazine.org/wp-content/uploads/2015/12/why-trust-in-theory06_1200-615x409.jpg)
@unpublished{Goodfellow-et-al-2015-Book,
title={Deep Learning},
author={Ian Goodfellow and Aaron Courville and Yoshua Bengio},
note={Book in preparation for MIT Press},
url={http://goodfeli.github.io/dlbook/},
year={2015}
}
No, our contract with MIT Press forbids distribution of too easily copied electronic formats of the book. Google employees who would like a paper copy of the book can send Ian the name of the printer nearest their desk and he will send a print job to that printer containing as much of the book as you would like to read.
This format is a sort of weak DRM. It's intended to discourage unauthorized copying/editing of the book. Unfortunately, the conversion from PDF to HTML is not perfect, and some things like subscript expressions do not render correctly. If you have a suggestion for a better way of making the book available to a wide audience while preventing unauthorized copies, please let us know.
Printing seems to work best printing directly from the browser, using Chrome.
It's difficult to predict. We're doing our best to finish it as soon as possible. We expect to deliver the manuscript to reviews by January, but there will be some delay with review, typesetting, etc. Please contact us if you are interested in using the textbook for course materials in the short term, we may be able to find a way to accommodate you.
Please help us make this a great book! This is still only a draft and can be improved in many ways. We've included some of our TODO marks to help you see which sections are still under rapid development. If you see a section that we appear to have finished writing and you find it unclear or inaccurate, please let us know. Please specify (by date) which version you are commenting on. Do not hesitate to contact any of the authors directly by e-mail: Ian (<lastname>@google.com), Aaron (<firstname>.<lastname>@gmail.com), Yoshua (<firstname>.<lastname>@umontreal.ca).
Older versions: