Quant Development

I have been using agile processes for mobile software development and find them useful, but challenging to get people change their thinking from waterfall approaches. Projects with many dependencies and large programs have challenges with agile processes

Written by Administrator Monday, 06 June 2011 12:29

Agile development processes - is there role for them when developing trading software?

I have been using agile processes for mobile software development and find them useful, but challenging to get people change their thinking from waterfall approaches. Projects with many dependencies and large programs have challenges with agile processes due to difficulty of coordination with many changes, but it works great with small projects or on top of mature platforms.

Do you use agile methods when developing trading/financial software and what kind of experiences have you had?
--
At my organisation we use agile development to the fullest extent. Our application is one of the highest performance exchanges in the world, plus all of the surrounding support infrastructure and technology.

Personally I think it would have been impossible for us to have delivered such a complex, high performance system without the benefit of agile development techniques. In essence our process is one of establishing effective feedback loops that allow us to always build on top of known good results. This is true in terms of the way that we capture requirements and deliver features as much as it is about how we create and deploy the software itself.

Personally I think that the sense that agile techniques only work for small teams or simple problems is wrong. I have worked on agile teams with more than 200 people engaged in the process of delivery in one role or another.

Don't get me wrong, small teams are nice, but the reason that agile processes work is not team size, it is because of it's focus on the scientific method - the Demming cycle at the heart of all agile processes of "Plan, Do, Study, Act" is a direct lift from the scientific method which can be written as "hypothesis"–"experiment"–"evaluation", and that is the most effective problem solving equation ever devised by humans.

--
The great benefit of agile is it's transparency without unneeded overhead. Most agile artefacts are simple, easy to create and use. Thus if you use agile drive teams should be easy.

The issue that some teams deny doing any architecture or reporting and call it agile. It's actually not. There are places for architecture phase in agile during refactoring and stabilization sprints. Reporting and KPI are also gathered throw burndown chart, storyboard, backlogs etc.--

My main development project (Apama CEP for FX aggregation and auto hedging) has been built by agile development. I am directly responsible for the P/L of this project and am not smart enough to give the team 6 months worth of requirements. I do not know where I want to be by then nor do I know which parts will give the greatest return for effort. So I give the team bite sized requirements and with a core team of 4 developers almost always have something delivered into production each week. Every 6 months or so we step back and look at something larger such as refactoring or a completely new module which will take longer to deliver. The project is a huge success and the developers are amazing. I couldn’t imagine this project being done any other way.

I see disagreements above but for me the size of the development team is a major factor in choosing agile development. I wouldn’t like to do this with more than 8-12 developers. Maybe this is because I also have a day job. If it’s a larger team you would need a full time project manager who deeply understands the business, the business requirements, the priorities and potential returns of each piece of work.
---
you can use bigger team, but think its easier / best for smaller teams with direct contact and delivery to the end users or Product owner. Apama development and Trading, seems to work very nicely with Agile, and the ability to give “stuff” to the user(production) also provides immediate feedback and allows quicker turnaround in functionality based on what was just completed. It also makes prototyping to a subset of users possible, and then has production feedback for the next release/delivery cycle. The Users can also adjust very quickly their requirements of what they need next, just as the market changes so can the priorities.--
I'm also a believer in agile but although agile allows early feedback to give direction to the project, it's still essential to have a core team of developers with a clear vision and thorough understanding of the 'problem domain'. You can not only rely on user-feeback Thus a team of agile developers for mobile are not necessarily a good team for doing agile in finance.
A big upshot of agile in finance/trading to me is that, since the trading domain is pretty mathematical but the mathematical concepts are not always intuitive to the users of the systems, agile allows to improve understanding and confidence in the system.

--
Indeed, exploring anything new successfully (e.g. handling fuzzy requirements, working toward an unknown final goal) depends on good quality feedback. The scientific method / agile approach makes obvious sense.

Our projects include a combination of the known and unknown. So, for example, we may not be clear what each message will do initially but we do know we need them delivered reliably. If we have the knowledge (from previous projects or the experience of others) we do not discard that in our agile project. We use it. This is also part of the scientific method, using existing theory where it is applicable.

For larger projects I have found it to be helpful to break them into smaller projects by drawing technical (e.g. API, core data entities) and organisational boundaries. Those boundaries are fixed relatively early and have to be determined by the most experienced developers of the team.

--
I've seen agile used to great success at financial institutions. I'm not a particularly strong proponent of agile (I have seen equally good projects delivered with waterfall), however finance has demanding users and regularly changing requirements hence regular deliveries, daily scrum / jira wall and good unit / integration test coverage really helps in this sector
--
• It has been 4 years+ since I have even come across a waterfall approach to development, and that project (a cash FX trading front end re-write for a big IB) overran horribly and was quickly deemed a failure. Agile has its issues of course, but I cannot get past  initial point about the Demming cycle; if you tighten the feedback loop you will write better software, period.

I do see how large teams can make agile difficult, but increasingly I feel this just requires investment in an extra leadership and co-ordination role. If you agree that a developer is more effective in an agile team (a debate in itself, but a premise I am completely sold on) then it falls to the good folk who hold the reins on the project to define the best team structures, reduce the communication frictions, and provide the vision and guidance required to ensure each sprint is heading in roughly the right direction.

--
Great comments. Thank you everyone. It is comforting to hear that so many people have good experiences from Agile-processes.

I still would like to go back to team and projects sizes. It seems that most of the teams have been quite small had worked with 200 people, which is quite large size for an agile. In my work we had over 1000 people in the multiproject setup using Agile-processes. The communication is crucial and we could not rely on tools. Especially when most of the tools are meant for individual teams. I do not have experience from Jira. Does it work with multiple cross depending teams using Agile?

On top of Agile methods we are still using traditional project management methods to handle different lead times in projects. Also dependencies between teams is generating lots of work. Do you have recommendations for dependency and cross team requirement management?

--
what is the nature of the 'lots of work' generated by the cross team dependencies? Is it lots of project management work? Too many Gantt chart re-jigs? Extra communication burden?

I was taught that you need to have three things in place to create an effective group working environment; People, Process, Technology. maybe you can break down each of these to identify where the weaknesses are?

People: Do you have strong people working in the correct roles at all levels of the structure? Can you identify the crucial roles, do you have your best people working in them? Have you allowed for absences and team changes?

Process: Have you defined the types of process you need to enable your people to be effective? Do you have them written down? Do your people find them useful or a burden?

Technology: What do you currently have to aid the people and support the processes? Is it effective? Does it create a burden in itself? Jira obviously can help here, but if you just throw a OOTB Jira setup at a large team they will likely hate it. You need to have the 2 P's in place, then configure Jira to support them, not force them to work to its regimen.

This may be obvious to you, but sometimes it helps to have someone prompt you to rethink the your approach from the ground up.

--
dependencies (other teams) for example might have repriorized their backlog or could be late with their work. We use some tools to track this, but in many cases scrum of scrum was used. In the beginning of the project time was spend to map the dependencies. We have been doing many projects (and programs) using traditional methods, so this time it was about changing them to match with Agile-methods.

--
• The challenges mentioned here are the usual ones the accompany almost all organizations in transition to Agile. So in my opinion the answer to the original question is definitely yes.
--
Sounds like you have differing levels of agile adoption across teams, that would be a tricky situation to manage I would think. What were your experiences with the scrum of scrums approach? Did you find it effective?---
sounds like you are moving in a right direction. In such a huge team agile will show it's advantages very easy. Having daily builds and short sprint iterations will help you to see risks earlier and stay on track.

A Preview of Storm - an exciting development in streaming Hadoop

Written by Administrator Monday, 30 May 2011 05:53

A Preview of Storm - an exciting development in streaming Hadoop

Here's a great description of streaming map/reduce from the engineering team at backtype.

(Please ignore the exchange between Nick and myself - we bring out the best in each other)

tech.backtype.com

--

Hi this seems great, can I get some more details to understand it.

Is something like S4?

--

So the word Hadoop is now becoming an alternative term for parallel processing. I think that would make more sense if the object being considered "Hadoop'd" specifically included a map-reduce construct as it's means of executing the parallelism and/or it leveraged HDFS and maybe Hive as it's default store.

But I digress ...

Storm is very exciting. I'm looking forward to it being available as open source. Is there an expected date for that?

Also, where might you open source it? If it truly were very Hadoop-like it would be cool to see real-time-streaming become part of the actual Hadoop ecosystem.—

Interesting, but this (based on the description) doesn't sound like anything that Distributed Programmable caches (Gigaspaces, Coherence, and Gemstone) haven't been doing for years, especially continuous streaming, automatic partitioning of messages, and RPC. It even sounds a bit more complicated. "Real-time" is interesting but what kind of latencies are we talking about. Gonna need more information before getting all excited. I don't see any in depth or API descriptions on the site. Hope to see docs soon!---

I agree with your assessment - so far. What I've found after 25 years in Capital Markets and then dabbling a bit in the world-of-the-web is that most of the firms out there just don't pay for software. So it's exciting to see this approach become so applicable that it might become open source.

Makes me wonder what's next on the 'software I'm willing to pay for front?!?'

We've been using these techniques for years on the street, either via custom code or platforms like those you mentioned (Giga, Coherence, Gemstone) and others. One area of direct comparison would be in the area of Complex Event Processing (CEP).

There are a number of firms that have applied the map/reduce paradigm there as well.—

Check out HStreaming at http://www.hstreaming.comfor streaming map/reduce right out of the Hadoop world.

Although the work that backtype did sounds cool and groovy, it doesn't represent a 'new primitive' as the authors claim.

Folks need to do a little more research before making bold claims.---

, it's called marketing. Hype your product to the point of lying... :-)

I'm a little concerned that everyone expects proprietary efforts to go open source. Companies invest millions to develop these products so they should be allowed to recoup their investments....—

I wanted to clarify what I said - it seems that the internet co's just dont like to pay for software. I agree with you wholeheartedly that firms should be rewarded for the investment. I have seen a lot of software developed in that space which has existed for years elsewhere. In capital markets, where technology is often an important competetive differentiator, firms are more willing to pay.

As long as it works. ----

Agree all around. I see three arguments for building. First, they feel they can't experiment at scale if they need licenses. I like "pay for production" and "pay for support" myself. That way I can play first.

Second, new developers in the Internet space were intimidated by enterprise tech. They didn't have time to learn EJB and the like, which was a disaster (thanks Sun & IBM [ironically]).Once they developed skills that diverged from main street the "religion effect" kicked in.

Third, misconception. How many times have I heard, "Microsoft won't scale". Well scale for what? I'd add to this that people don't count the manpower costs so much as the licenses. For instance I like using a platform like "Vertica" for analytics. It's fast, reliable, and scales huge. It's all SQL with extensions. No java, no C++, easier to install and scale than Hive, Cassandra, HBase; fully supported. How much do I save on manpower costs not just to build but to maintain and correct, not including the "Certainty Premium". I know it works. (Sorry if that sounded like a commercial just trying to make a point).

--

There are also some good products out there like VoltDB (in mem DB) that are open source. You only pay for the Enterprise GUI and for support. I believe Gigaspaces still has a "pay for production" and "only after you start making money" policy.

----

• I am not really familiar with the Storm's architecture (for obvious reasons) but this is for sure not a million dollars investment. Taking into account that all basic components for building distributed real-time systems are already open-sourced: ZooKeeper as cluster coordinator, any Open Source message broker (HornetQ, RabbitMQ, ZMQ) as a low level messaging system, the only thing one need to implement - distributed Executor service and set of distributed coordination primitives on top of ZooKeeper. And you can make announcement. HA , fault tolerance support will require most of the development and testing time, of course

===

As a person with a "web-background", I can say the world of web2.0 tends to build systems on their own. Interestingly, they not necessarily do that because of a lack of otherwise options. Instead, for the companies I have interviewed, opensource has a strategic dimension. Web2.0 companies have a deep and strategic will do be able to do everything in-house. Sometimes wheels get re-invented in that process. Might be, Storm is a candidate for that.
In any case, although this is not always the best decision from an expenditure point of view, mostly I honor this attitude. With every additional project that re-invents the wheel, the state-of-the-art is advanced, new insights can be drawn and lessons learned from such endeavors.

--

: I agree with you generally, but I've also seen a lot of large 'Web 2.0' co's, who can spend the money not spend it because 1) they don't know the solution is already out there, and 2) they're just cheap.

==

I think the important take away here is that scatter/gather, map/reduce, distributed RPC, etc is becoming more widespread. As internet companies push towards ever lower latency, they'll discover tricks and techniques that have been deployed for years on Wall St, Telephony, Signals Processing, etc.

It's fun to watch

Need Trading System Testers for a Forecasting Tool based on Probability

Written by Administrator Tuesday, 24 May 2011 15:56

Need Trading System Testers for a Forecasting Tool based on Probability

Hello, I have been working on a forecasting model that is based on probability. I originally developed the model in 1989 and have done some enhancements since. I took a vacation from the markets for 10 years, and got the itch again in 2006. So off and on since 2006, I have been working on a forecasting tool and I'm about 4 weeks away from deploying it for testing. This model forecasts the direction of the market for S&P's, Bonds, and few other markets. I am looking for a few traders to look it over. Does anyone have any suggestions on traders that might want to look at a probabilistic forecasting model?

 

Quan development: Why you dn't use Ninja Trader

Written by Administrator Monday, 02 May 2011 13:37

I have been involved in this business about 60 years. There have been about three programs that were written by programers in that time that sucked. Ninja is about number 2. It's success is based on it being free until you actually trade or the broker pays for it. I know ninja users the beginners and a few professional use the program, I will offer it to others in an 80 trader room for the public. Most professionals do not use charts or indicators. Ninja is not user friendly that is why there are classes everyday on how to use it. One of the best programs written was windows on wall street it was purchased by TradeStation because it could have put their platform out of business. It would be easy to fix Ninja and make it user friendly. How complicated does it have to be to buy and sell a security? You are one of the reasons that Ninja is screwed up because you do not realize the problem. I think that if someone had to use all the features of Ninja they would be spending their life emailing for help. I have a friend that uses Ninja among others and there is always a problem. If management had any idea of what they were doing they would offer a variation of the program that was user friendly.

--

so would you categorize yourself as preferring the shotgun approach to problems?

As to NinjaTrader, I would suggest that those who only need to buy or sell a security without interacting with external elements as allowed for by C# and .Net do not need the power and therefore the complexity of NinjaTrader.

I do agree in many aspects NinjaTrader is not overly user friendly but neither is Quant Studio, a native C++ program using FIX or any of a dozen other high end professional programs, including the API's of most brokers (IB certainly comes to mind).

You are definitely correct as to me not realizing the problem; I don't see a problem with NinjaTrader. If I want charts, I use TradeStation, if I want speed of development coupled with a reasonable number of features, I use NinjaTrader and if I want absolute control and power at the cost of an extended time to deploy, I use either the broker's API or if portability is required on top of all that, I use FIX.

You have to choose a platform on what is right for your objectives, resources and skill level.