From this point through the 1800s the European powers settled in the regions of Southeast Asia, but for the most part left Brunei alone. There were a couple conflicts with the Portuguese as the people regularly fought with the Spanish. The Spanish slowly took numerous lands in the region, including the city of Manila (in the Philippines) in 1571, which was at the time controlled by Brunei and Brunei itself in 1578. The Spanish soon after left though, returning to the Philippines.
What the Spanish did do was destroy the trade that Brunei took part in as the Spanish and other European powers began to control the seas. From this point into the 1800s Brunei was a small sultanate that lived on trade, which was weakened and didn't recover for centuries.
In 1839 independent British explorers arrived to the region to help put down a revolt. This led to further British intervention as one of these explorers sought to take the region as his own. This led to Britain getting officially involved as they entered the region in 1846 and took Brunei Town. This led to the forfeiture of lands to the British, a trend that continued as the sultans of Brunei lost all of Sarawak and eventual led to the region becoming a British protectorate in 1888.
During World War II the Japanese invaded and took the region from the British, but the region was liberated by the Australians in 1945. Despite this the British continued to control the region until 1959 when the sultan of Brunei gained control over domestic affairs and in 1984 the sultan gained full sovereignty over Brunei.