The original Dudley Castle, a simple wooden motte and bailey, constructed in 1070 by Ansculf de Picquigny, was rebuilt in stone in the 12th century. Ansculf de Picquigny was succeeded by his son, William Fitz-Ansculf, who in turn was succeeded in the 12th century by Fulk Paganel who possibly married Fitz-Ansculf’s daughter, Beatrice, but there is no direct evidence of this.

The Paganels built the first stone castle on the site. During the civil war between Matilda and King Stephen, it was strong enough to withstand a siege in 1138 by the forces of King Stephen, but when Gervase Paganel took part in the failed rebellion against King Henry II in 1173, which lasted 18 months, the castle, along with about twenty others was demolished by order of the king. During the rebellion, many towns were destroyed and a large number of people lost their lives. In 1165 Gervase, founded Dudley Priory.

In 1194, Gervase Paganel was succeeded by his nephew, Ralph de Somery, who was the son of John de Somery and his wife Hawise, sister and heir of Gervase Paganel. Ralph did not get full ownership of the estate until the death of his mother in 1208, who had inherited some of the land herself.

In about 1205, Ralph swapped land he held at Wolverhampton with King John, to obtain estates at Kingswinford, including Pensnett Chase which was used by the Lords of Dudley as a hunting ground and later proved to be a rich source of coal and other minerals.

Ralph, who died in 1210, married Margaret le Gras, niece of William, Earl of Pembroke and had three sons, Ralph, William Percival and Roger, who successively inherited the barony. Roger began rebuilding the castle in 1262. When he died in 1272, work on the castle was still underway and it was completed by his son and successor, Roger 2nd, who took an active part in the campaigns against the Welsh. He died in 1291, when his son John was just 12 years old. He was survived by his widow, Agnes, another son, Roger and two daughters. In 1303, John de Somery took part in the Scotch war and was presented with a knighthood in 1306. He continued to take part in the Scotch wars and immediately after the battle of Bannockburn was summoned to Newcastle to help in defending the north of England.

John was very unpopular with his tenants and neighbours because of his arrogant, rude and dictatorial way. He also extorted large sums of money from them. It seems likely that because of this, Nicholas de Somery, son of Robert de Somery and others were accused of breaking into Dudley castle in 1321 and removing £1,000 and goods worth £200. It is believed that the castle had been completed by this time.

John de Somery was the first person in his family to be summoned to Parliament as a baron. He died without children on the 29th December, 1321 and his estate, including the castle became the property of his eldest sister Margaret, who married into the de Sutton family. The Somery's often used the family name Dudley.

At this time large amounts of building work took place at the castle, including a barbican to the outer gatehouse and chapel. Parts of the castle that were built around this time, still survive. The keep, which was built on the Norman motte, consisting of core of limestone rubble encased in clay, had four drum towers, one on each corner, 9.8 metres in diameter.

Most of the main gatehouse dates from the 13th century. The double gateway with two portcullises was constructed at this time. The barbican was added to the outside of the gatehouse and the whole structure became known as the 'Triple Gate'. It was originally connected to the keep by a thick curtain wall. When built, it had three floors with machinery for operating the portcullises on the first floor and a guard room on the second floor. Above the guard room were the battlements.

John and Margaret de Sutton were only in possession of Dudley castle for a few years. John was accused of being involved in Thomas Earl of Lancaster’s rebellion against King Edward II and was imprisoned in the Tower of London and forced to give up his right and interest in Dudley Castle, the town of Dudley and the manor of Sedgley, to Hugh le Dispenser, a favourite of the king.

In January 1327 the king was ousted by his wife Isabella and her lover Roger Mortimer. Later that year, the castle was returned to John and Margaret.

Their son, Sir John de Sutton 2nd, born in 1310, became the first Baron Sutton of Dudley. He married Isabella, daughter of John de Cherleton, Lord of Powis. It seems that John de Sutton mortgaged the castle to John de Cherleton, to whom he owed £3,000.

In 1330 to 1331, a number of people were summoned before the King’s Bench, including William le Fisshere, charged with besieging Dudley Castle. They besieged the castle for two days and assaulted John de Cherleton. A similar thing took place again in 1331 when the castle was besieged and items were taken away. The besiegers were led by Joan de Botetourt, daughter of Roger de Somery and sister of Margaret de Sutton.

John de Sutton had recovered the castle by 1337. On his death in 1359, the castle and his estate became the property of Isabella, until she died in 1397, when it was handed down to her great grandson, John de Sutton 3rd, who had been born in 1339. John the 3rd married twice, first on the 25th December 1357 to Katherine de Stafford, then after her death in 1361 he married Joan, daughter of Sir John de Clinton of Coleshill.

On the 6th December, 1361, John 3rd and Katherine had a son, John 4th, who became the 3rd Baron Sutton of Dudley. John 3rd died while his son was still young and so 350 marks were paid to Sir Philip Le Spencer to be John’s guardian and also to arrange a marriage between his daughter Alice and John. Alice died in 1392 and then John married an unknown lady called Joan, who died in April 1408. John 4th and Joan had a son, John 5th, who became the 4th Baron Sutton of Dudley and married Constance Blount, daughter of Sir Walter le Blount of Barton, who was killed at the Battle of Shrewsbury.

John 5th and Constance had several children, John 6th, Thomas, Elizabeth and possibly another son called Humphrey. John 5th died on the 29th August, 1406 when John 6th was six years old. Constance’s mother Sancha de Ayala took custody of him and was granted his estates.

From an old postcard.

The Growth of the Town

Although the castle played a large part in the town’s history, the town itself developed entirely to the south of the castle, possibly as one long street, where the High Street now stands, between the churches of St. Thomas and St. Edmund. As the population increased, other streets appeared to the south, east and west.

Little is known about the early town, which would have been planned and developed by the ruling family, possibly Gervase Paganel in the latter half of the 12th century. An important part of life there was the market, roughly in the middle of the village. There were no shops at the time so that anyone wishing to buy or sell items had to use the market. King Street and Tower Street were built alongside and linked by interconnecting streets with properties and land that was rented from the ruling family. The layout possibly follows the boundaries of the old ploughed arable fields that were slowly disappearing as the town increased in size.

Stone Street Square was originally called the Horsepool because it was the site of a pool that was fed from a spring at the back of the market place, between Stone Street and Wolverhampton Street. It was called the Horsepool because horses were watered there.

The market place was mentioned in 1261 when Roger de Somery sued Giles de Erdington, Dean of Wolverhampton, for opening a rival market there. An agreement was reached so that Dudley people could use it free of charge.

In 1339 mention is made in a deed, of a high market cross, standing at the northern end of the market place.

A view of Dudley market place at the turn of the 20th century, showing the relationship between the early town and the castle. From an old postcard.

The medieval market was mainly the place where agricultural products were bought and sold, but there were also some small metal working industries in the locality, thanks to the plentiful supply of local iron ore, coal and wood, so locally made metal goods would also be available. By the late 13th century, iron was smelted on the thickly wooded Pensnett Chase on the southern side of the town, which like much of the surrounding woodland was mainly used for deer hunting. The area of land on the north eastern corner of the town, to the east of the castle, stretching northwards into Tipton, called the Coneygre (meaning rabbit warren) was where the Barons bred large numbers of rabbits, having received the right to use it for this purpose in 1247. Castle Hill was previously called Coneygre Hill.

In 1476 much of the town centre, including the old wooden St. Edmund's Church, was destroyed by fire, which would have quickly spread through the half-timbered buildings with thatched roofs. It is not known how far the fire spread.

Dudley soon grew into a flourishing little town with an expanding population and a network of trackways providing easy access to the surrounding towns.