MLDS - All the Webhost You Will Ever Need

Attention Ovid-Elsie Alums.

During the last O-E playoff game WOES ran out of space on their streaming server for people out of town to listen. I (via MLDS) would like to provide WOES with the extra server, software and bandwidth to support an unlimited number of listeners.

What do you think? Please comment, let me know:

• Did you try to listen but couldn’t connect?
• Do you want a stream other than Real?
• Do you know anyone who wanted to listen but didn’t know you could online?
• If you would listen from my stream!

If you are a O-E staff member, student or parent. Please ask O-E Schools and WOES to accept my offer. Or if you have any ideas please place them in the comments. If you wish to email me directly my email is brockp@mlds-networks.com

Brock Palen, O-E Alum Class 2002

We use Gold an allocation manager built by Cluster Resources to collect data on ran jobs. We don’t currently use it for allocation management.

While Scott Jackson did a great job creating gold, it was not built for speed. He backed Gold’s data store with the PostgresQL (PSQL) database but did not create any indexes on the database. This caused PSQL to run very sub optimally. Once Gold started showing some performance problems back a year or so ago we put some work into speeding up gold. This work was added to Gold 2.1.5.0. This solved our problems for a while, till now.

Gold was running a large number of quires of the form:

SELECT g_reservation_allocation.g_id,g_reservation_allocation.g_amount FROM g_reservation, g_reservation_allocation
WHERE ( g_reservation.g_id=g_reservation_allocation.g_reservation AND g_reservation.g_start_time< ='1226009628' AND g_reservation.g_end_time>‘1226009628′ AND g_reservation_allocation.g_account=’2′ ) AND g_reservation.g_deleted!=’True’ AND g_reservation_allocation.g_deleted!=’True’;

This query even with index’s on a few of the WHERE conditions PSQL was doing a sequental scan. Using the ‘EXPLAIN ANALYZE’ syntax we see:

 Nested Loop  (cost=0.00..53411.23 rows=1 width=12) (actual time=2878.228..18789.857 rows=502 loops=1)
   ->  Seq Scan on g_reservation_allocation  (cost=0.00..47592.58 rows=1657 width=16) (actual time=2870.689..17636.409 rows=928 loops=1)
         Filter: ((g_account = 2) AND ((g_deleted)::text <> ‘True’::text))
   ->  Index Scan using g_id_idx2 on g_reservation  (cost=0.00..3.50 rows=1 width=4) (actual time=1.238..1.239 rows=1 loops=928)
         Index Cond: (g_reservation.g_id = “outer”.g_reservation)
         Filter: ((g_start_time < = 1226009628) AND (g_end_time > 1226009628) AND ((g_deleted)::text <> ‘True’::text))
 Total runtime: 1707.245 ms

So even with the indexes PSQL is not optimizing the where condition. The worse being the check for !=g_delted.

As you can see an 1.7 second query is way to long, if we want to start 100’s of jobs a minute. To speed this up we found that PSQL lets indexes have WHERE conditions. These conditions lets PSQL know right away where to find data where g_delted!=’True’.

create index  g_reservation_acct_idx ON g_reservation_allocation (g_account) WHERE g_deleted!='True';
create index g_id_where_idx ON g_reservation (g_id) WHERE g_deleted!='True';

The first index lowered the query time to around 16ms. The second index is chosen over a regular index just because it is a smaller search area. It was really added for another common slow query that Gold does. Thus it only speeds up the above query to about 8.5ms.

 Hash Join  (cost=1462.31..1546.42 rows=1 width=12) (actual time=3.846..7.951 rows=502 loops=1)
   Hash Cond: ("outer".g_reservation = "inner".g_id)
   ->  Index Scan using g_reservation_acct_idx on g_reservation_allocation  (cost=0.00..75.85 rows=1649 width=16) (actual time=0.020..2.175 rows=928 loops=1)
         Index Cond: (g_account = 2)
   ->  Hash  (cost=1462.02..1462.02 rows=117 width=4) (actual time=3.763..3.763 rows=502 loops=1)
         ->  Index Scan using g_id_where_idx on g_reservation  (cost=0.00..1462.02 rows=117 width=4) (actual time=0.134..2.876 rows=502 loops=1)
               Filter: ((g_start_time < = 1226009628) AND (g_end_time > 1226009628))
 Total runtime: 8.492 ms

So our performance per/query went from 1600ms to ~10ms. Not bad, now we can start 100’s of jobs per minutes. So why was this performance so much better? The reason our performance started to become a problem the second time around was due to how many jobs we had run. g_reservation_allocation had 1,649,899 records. Not huge but big enough that if your doing a SecScan like PSQL was doing running over a gradually growing table would get slower and slower. Having an index of only valid jobs (g_delted!=True) gives PSQL an oprotunity to know exactly very quickly where to look on disk. Of which there was only 928 records. This number will only change with number of running jobs. Which will grow over time as we add hardware, or we run more serial jobs. In any case it will grow much slower than the total number of jobs ever ran. We should hit 2 million records very soon.

Moral is, if your using Gold add the above index’s. It will help you out greatly. You can find my email notifying Gold Users here: http://www.supercluster.org/pipermail/gold-users/2008-November/000123.html. I hope Scott adds this index to the next release of Gold so you don’t have to add it your self.

You can always email me questions at: brockp@mlds-networks.com

First watch this from TED

Now those of you who know me will pick out right away the two topics that stick out to me in this video:

• Why did it take so long to happen?
• Why didn’t it happen when Government had a monopoly in space?

Why did it take almost 50 years after we went to the moon for something like this to happen? When we finally did make it to space why did it take just a small group of people who wanted it to happen? Is there something fundamental about our old structure of access to space that fundamentally made it impossible?

My personal reason for why the cost of going to space never went down was that there was a feeling that space was owned by the government. If our country isn’t socialist on the ground it is in space. If you thought trying to get all the permits to build a house was bad talk to someone from this project about getting Ok’d for going into space. People were just not allowed in two ways, there was a paper wall, and people just thought it was NASA’s job. NASA’s job is to study earth and space, not to exploit it for our use on earth. I think people thought NASA’s job was to do it all when it was not. Even though it took so long I think it was good that it is not NASA offering flights to space for average joe (well above average income joe, but not richie rich).

NASA even if the people thought that it was NASA job to use space for our benefit, NASA could not. Work for a public employer and Milton Freedmans, “anything the government does, private sector can do for half the cost” you will know why NASA can not. NASA will not have done such a thing even if it wanted to. Look at the Aries project to replace the shuttle. Late and over budget, yet offer of $10 Million in cash in a few years you can ride to space for $200k. Amazing, government (in this case NASA) couldn’t/wouldn’t do it, so regular Americans did it. Yes that is the power of free people in a free country. We can find ways to do things better and suffer the loss of failure and gain of success.

Look at the numbers, you can fly to space for 200K/person. If a person weights around 175lb on average, that is a cost of $1142/lb. Compare the shuttle of $10,000/lb. Well the shuttle is 20 year old right? Yes and we knew it cost more than the Saturn V when it was built, “but it was a good idea” (wrong) and we did it anyway. See government does not have to live with failure like someone doing this on a shoe string. Yeah $10-25 Million isn’t your normal string but compare that to Aries and show me what is efficient Oh and notice how he says at the end that $20,000/flight is on it’s way. I think its time for NASA to just pay to have someone else (the market) to fly their stuff to orbit. The market has shown how it can built and fly such a device in less than half the time for less than 25% the cost and operate for even less.

So the next time someone says thats important the government should provide that. Think back to how space flight was when it was government controled. Over priced, wasteful in people (think what all those skilled people could be doing), resources (all that fuel not being blown all over Florida), and availability. If they did get the cost down to $20,000 and I wanted to avoid buying a house for another few years I could fly to space, and I don’t make that much.

My what a few people can do. Just proves If you think something should be done go and do it, don’t wait for the government to do it! Get off your butt, find others who think like you and put it together. Or when your old, you will still be waiting.

I was recently tasked with resizing a large group of images to post online. The original images were to large both in file size (to long to download over dial up) and physical size (images went of the edge of screens).

To do this I went to my trusty command line and used www.imagemagick.org. Image Magick comes with the wonderful convert tool. Using convert I have made jpg’s into gifs and mtiff’s into gif’s. But I have never used to resize and compress before. Lucky me there is a huge list of options. I even thought about watermarking the images (copyright info) using convert but decided against it.

To start I figured out how to resize then how to compress. The final command line option I came up with was:

convert input.jpg -resize 50% -quality 50 output.jpg

I then proceed to do this in a shell script, which took a few minutes to run, but still hours faster than I could ever do with photoshop or the GIMP.

#!/bin/bash
for x in `ls`
do
   IMG=`echo $x | awk -F. '{print $1}'`
   echo $IMG
   convert $IMG.JPG -resize 50% -quality 50 ${IMG}_bp.jpg
done

Now there is a bunch of resized images ready in minutes. The input images were all of form FILENAME.JPG the output were all the form FILENAME_bp.jpg

Questions about hosting images should go to brockp@mlds-networks.com.
The images can be found at the: Michigan Jersey Cattle Club

Some thoughts on government intervention that I am sure most people at face value would say is great really distributes wealth from the lower and lower middle classes to the upper middle and high class.

It was all done with Hybrid cars. Turns out our all wise government decided we need to have tax credits when buying a new hybrid car (Reference). At face value sounds great, this will encourage production of hybrids because it will lower the cost to get a hybrid thus increasing the number of people who can afford them. It gets support from green/auto/union etc.

What is the side effect of this policy though? Turns out in absolute terms it’s not all great. Hybrids being new and complicated are kinda pricey. Cheapest base model I found was around$20,000 (2008 Toyta). This is out the range of price for a regular house hold. Remember the median house hold income is around $47,000/yr and that’s on average a family including kids. So the only people who can really afford these cars are upper middle class and wealthy people.

So this is where the short change comes in. Only wealthy people can afford these cars and thus they get the tax break. Everyone pays taxes though. So everyone pays in to provide these breaks but only those who are rich can get them.

Now there is some technical arguments against this, mostly based that we run a deficit and thus the whole does not make up for these credits, its just financed. In general thought though when was the last time you saw someone say we could raise the taxes on the rich (roll back the Bush tax cuts) also say and get rid of that tax breaks on hybrids. I best most who want to do the first would not want to do the second. In the end it’s all money.

Just trying to keep people honest. If anyone can think of a reason why we should do such things please try to convince me.

Ok random thought time. What do we spend 1/3rd our life doing? We engage in an economy. We work, we buy, we sell. When I sit back and think about it of the time we spend living engaging in some for of the national economy is quite large. I also think its the one thats most fragmented. I don’t think people think about the connection between work, buying, and freedom.

In the last 50 years or so the government has had more and more control over our lives by getting involved with the economy. Organizations like Fannie Mae and the FDA in their own way intended to help us have caused problems all their own. I am now going to regurgitate thoughts onto paper so hang on.

How does Fannie Mae help us? How does Fannie Mae hurt us? While I think Fannie helps some with keeping the cost of owning a home lower (see my older post, about how I don’t know if this is a good thing or not). The problem it can cause is worse than the benefit. Many of these organizations are put in place as though they can never fail. Problem is anything built by man can fail and does given the right means. Problem is when a government entity fails it can threaten to really hurt a market. Notice I don’t say take down. You can every kill a market, as long as there is two free people each with something the other wants the market will work. Its more a question how well it would work.

Take large banks. Right now we see many of them having trouble because they got over their head with sub prime loans, just like Fannie Mae did in the name of ‘promoting home ownership’. Not all the banks did though. There are many banks, many very large when one of them fails it hurts. It is not possible for all banks to fail, there is just to many. So even in the worst kind of misbehavior the damage is limited. Now lets look at similar government organizations. While Fannie Mae and Freddie Mac both service the same purpose they are only two institutions and its much simpler for 2 to screw up and fail than many. Its just the statistics of it. We now find our self’s where the government has to bail them out at a huge cost. The bottom and consequences because Fannie Mae exists is worse than if they were never created in the first place. Without them blood would have been split, oh the initial shock might have been greater but how far the market could fall should government not do anything would be much less.

Because government got involved in the private property business, that is encouraging the owner ship of one type of propery (real estate). The problem can and are worse. Then the amazing thing happens that only happens in government which I will go on to explain next.

All institutions have intentions and results. The hard part is getting from the first (intention) to the second (results) and having them be equal. In the market if you get to far from results equaling intention you go under. People loose there jobs (that’s great) and the organizations that do make it from intention to results grow. Those who worked at the now failed organization are freed to work in more effeicnt and successful organizations. That is the market is self correcting though it is not kind. In the end we are all better off.

Government instutions also have intentions and results. The change here is that when the results don’t land near intentions instead of failing more often money and resources and authority (new power invoked by congress) in added making the failed beast even bigger. This is what amazes me. Fannie Mae screwed up, is put its fingers into the wrong pot and got bit, in a working market they would be eaten up by those more prudent and better heads would lead from then on. Instead the same people will be taking money from tax payers (who are now bearing the risk of a publicly traded company no the share holders). To keep them up and even lax the rules on what Fannie is allowed to buy! Does this make any sense?

Fannie is just one example, there are many more, find one time where a government organization has failed and the leading body, be it congress or local board, say “shut it down, it failed”. No we keep trying with the same broken POS.! Tear it down, maybe try something radically new but judge them on their results and admit failure. I think history shows more often failure by such organizations than success.

In the end, again I feel my brain pushing me more and more towards Ben Franklyn style thinking. It is true I think, that the Libertarians are the closest thing we have to our founding fathers. If I was ever asked if I would rather be a man like the founding fathers or the current crop of leaders, my answer would be quick and simple.

End Rant.

Not all the time, just every time I try to do something it fights me. I was asked to create some videos like I have done in the past. Problem is to deal with the heating problems Apple decided to just crank the speeds up on the fans. So now even sitting idle my fans run at 4000-5000RPM and doing anything they jump to 6000. Point being my microphone (used to dub voice over the videos) is picking up all this racket. New mic in my future.

/rant

This vacation I spent a few days with the people of the Marine Mammal Center. I will defer to their website to see what they do. In general this is a non-profit that goes out and saves/rehabs sick/injured marine mammals. I was lucky enough to engage in both the care of animals currently in their care yet to be released and pick up a newly found stranded Sea Lion.

Now those of you who know me know I am a big believer in small government and that most (if not all) giving should be done by private organizations funded by private people and staffed by private people.

All I can say is the MMC is a wonderful example of this. This organization if fully (short 3% provided research grants) funded by private donations. The MMC is very efficient. They have to be, there isn’t any magic money being taken from tax payers. Because of this they have to show results. They have a wonderful public education program which probably saves more seals from mistakes made by the public than their active recover saves each year. They have a very active community of volunteers many of which have been there 5 years or more. If you have every worked in a like organization you will know that keeping people around so long is very hard. The MMC is doing a wonderful job of this. They also do a great job training new people in the care of like animals to do this else where.

In short the MMC does its job very very well. Better than any similar case that relies on government money or keeping a single Senator/Politician happy. They respond to their own wish to see the animals thrive and work to do that in the be way they can.

It is to me a model of the way charity in the United States should be done. Private people working towards their private interest in protecting and helping these animals. They have a wonderful new facility being built that is fully funded by private persons that will enable them to help yet more animals. They are grateful for these people who have provided the funds and are also grateful for all those who while they may not donate cash donate their time and their backs.

Just a wonderful example, wonderful. I have never worked with any government organization that could even compare to how far a dollar or man hour goes. I would say they are at least 3x’s as efficent as the next government ran program.

Everyone at the MMC thank you very much and for putting up with my questions. I wish you the best in your efforts, and yes I think the Elles are very cute and lazy.

Remember they run on private donations do what you can: Donate

Source of income for operations data:
http://www.marinemammalcenter.org/about_us/financials.asp

I am currently taking a class to get my Michigan Motorcycle endorsement.  The class is located at Washtenaw Community College. The class costs only $25 lasts 6 days (Monday-Thursday then Monday,Tuesday), first day is book work the following 4 days are all on the range on bike they provide the last day being a written test and skill test.

It is a great value, cost includes all materials and gas.  You will need geat to protect your self.  On this they are very strict but they do have some helmets you can borrow.

I have to recomened this class to anyone who would like to learn to ride.  I am not even done yet, or sure if I will buy a bike, or if I will even pass!

 Here are open classes:

WCC Lifelong Learning

So Travis leaves this summer to work for Microsoft so I will need someone to take his place.

Its a two bedroom apt. Fully furnished though you can bring your own stuff.  Ground floor, AC, Grill, dishes, pool, etc.

Comes with a parking spot. 

Water and TV in included.  I’m asking $450/month.

Its on North Campus so great for Music school or COE or any Ann Arbor native.   I work here year round so you never have to worry about a subletter if your here in the summer because I will be here also.  If you need you can take over mid summer after travis leaves but the term is September to August.

If you contact me soon (before March) the rent can go down so act fast.

brockp@umich.edu

or

1(989)277-6075